When I import an HTML file, all references to   are changed to a question mark. See paragraph 3 on the attached file that begins with "Our site features".

CommentFileSizeAuthor
index_0.html4.88 KBrgraves

Comments

dman’s picture

dman commented

When running with htmltidy via commandline, the setting:
numeric-entities: yes
Is correctly translating   into   and all goes well.

According to the paramters passed to the extension (import_html/coders_php_library/tidy-functions.inc line 49)
It should be doing the same.

... my current test setup isn't currently using php-tidy, but I'll see if I can test it.

rgraves’s picture

rgraves commented

I haven't been running it from the command line. Not sure how to. I've been running it via the web interface in Drupal.

rgraves’s picture

rgraves commented

Is there a way to run the whole import html module from a command line instead of through the web interface? I'd prefer that since our large site can cause the web interface to timeout easily.

dman’s picture

dman commented

Um, HTMLTidy is a third-party executable, available as a commandline tool, or a PHP extension library.

Import_html tries to use the built-in extension, if available. If not it tries to execute 'tidy' on the server directly.

I can't think of a good way to run Drupal and import_html direct from commandline, sorry.
I was meaning I tested the phase of importing where the web interface runs tidy - it takes in your entities and returns numeric entities (which don't turn into ? marks)

To debug at your end, did you try turning on the debuglevel in the top of the code? Set it to 3 and Just test on 1 file. You should see exactly where the text goes wrong, if it's due to tidy, or one of the other transforms that happen.

rgraves’s picture

rgraves commented

I just did a large import and I received the following errors

At the top of the page:
----
line 6671 column 66 - Error: is not recognized!
line 6672 column 39 - Error: is not recognized!

: in debug_pre(), line 95 debug.inc : in xml_tidy_file(), line 155 tidy-functions.inc : in xml_tidy_string(), line 76 tidy-functions.inc : in parse_in_xml_file(), line 69 xml-transform.inc : in _import_html_process_html_page(), line 1705 import_html.module : in _import_html_import_files(), line 1513 import_html.module : in import_html_import_files_page(), line 1252 import_html.module : in menu_execute_active_handler(), line 418 menu.inc 19.80s elapsed. (1980 total)
----

and then after it said it inserted all the nodes:
----
* user warning: HTMLTidy failed to parse the input at all! It's probably very problematic HTML. A working version of tidy IS at /usr/local/bin/tidy isn't it? I ran /usr/local/bin/tidy -q -config /usr/local/lib/html/drupal/modules/import_html/coders_php_library/xhtml_tidy.conf "/var/tmp/htmSaaWAi" and it returned: 2 in /usr/local/lib/html/drupal/modules/import_html/coders_php_library/tidy-functions.inc on line 156.

* warning: DOMDocument::loadXML() [function.DOMDocument-loadXML]: StartTag: invalid element name in Entity, line: 1 in /usr/local/lib/html/drupal/modules/import_html/coders_php_library/xml-transform.inc on line 96.

* warning: DOMDocument::loadXML() [function.DOMDocument-loadXML]: Extra content at the end of the document in Entity, line: 1 in /usr/local/lib/html/drupal/modules/import_html/coders_php_library/xml-transform.inc on line 96.

* user warning: Failed to parse in xml source. [files/education/Spring2003.html] in /usr/local/lib/html/drupal/modules/import_html/coders_php_library/xml-transform.inc on line 97.

* Failed to initialize or parse XMLdoc input
* Failed to process file '/Spring2003.html'
----

The first warning indicates that HTMLTidy didn't run which would explain why it's not converting  . I don't recall seeing that error for other imports I've done however. Do you have any ideas why HTMLTidy might not be running based on this error? It is located in /usr/local/bin/htmltidy as the script says.

Also, at the top of each imported page, I get the following comment:

Unlike the first set of errors above, this error has occurred every time I use Import HTML. With the exception of this comment, the page appears to have been imported into Drupal successfully (i.e. navigation, header, footer, etc have all been stripped from the original source).

dman’s picture

dman commented

What this means is that HTMLTidy didn't run SUCCESSFULLY on that particular input, and was so confused it was unable to tell me why.

There exist a few HTML files that are just so nasty even HTMLTidy cannot grok it.

if you run 

/usr/local/bin/tidy -config /usr/local/lib/html/drupal/modules/import_html/coders_php_library/xhtml_tidy.conf Spring2003.html

You may see what the problem is.

The other few problems are just cascades from that failure, as the parser attempts to read the bad file that tidy rejected.

Not sure about the 'is not recognised' messages. They will be from the XML reader and probably mean & entity problems.

dman’s picture

dman commented
Status: Active » Closed (fixed)