When the original documents contains < and > these are always converted to the '<' and '>' symbols, hence producing incorrect html output. It should leave these tokens as-is.

For an example, see this document and search in the resulting 'Unfiltered HTML' source for the text "../configure", the preceeding option "--with-" is printed while the original HTML file contains "--with-<target>"

It seems that it's not 'tidy' doing this because it's not happening on a manual run.

Note: I had to set in xhtml_tidy.conf "input-xml: yes", otherwise, the anchors are broken in most(?) browsers. This option does not influence this issue though.

Note2: this was discovered using the 'quick demo' option.

Comments

sspr’s picture

sspr commented

Oops, forgot to use the 'code' tag. The report should have read:

When the original documents contains &lt; and &gt; these are always converted to the < and > symbols, hence producing incorrect html output. It should leave these tokens as-is.
For an example, process this document and search in the resulting 'Unfiltered HTML' source for the text "../configure", the preceeding option "--with-<target>" is printed while the original HTML file contains "--with-&lt;target&gt;"

dman’s picture

dman commented

It's not Tidy, it'll be the XML parser. I've seen that happen before, not with this, but in historical XSL wrangling.
Without a certain directive (can't quite recall what it is right now) the tokens get interpreted as literals, and fed out again as what the XML parser thought they were.
Um....
Annoying.

dman’s picture

dman commented

...however, I just did a test run on that very URL, and it came out bloody excellent. No problems at all.
Thus it's probably up to the version of XML parser.
Under :
PHP Version 5.1.2
libXML Version 2.6.24
libxml2 Version 2.6.24
libxslt Version 1.1.15

It's all cool.
I'm well aware that older PHP4 XML libraries were handicapped, so I'm not sure if that's what you'd need to look at to find the problem.
Worst case scenario os a pre-parse replace of problem characters, and substituting them back in again later. Boring.

sspr’s picture

sspr commented

Thanks for this quick response. I thought Drupal did not run on PHP5.

I am running:
PHP 4.3.11
xsltproc 1.0.16
libxml 2.6.2
libxslt 1.1.0

On a Debian woody. My provider is moving to Debian 'sarge' and PHP 4/5

dman’s picture

dman commented

I just tried the same page on
PHP Version 4.4.1-pl1
DOM/XML API Version 20020815
libxslt Version 1.1.15
EXPAT Version expat_1.95.8
Sablotron Version 1.0.2

(PHP4 libraries are altogether different) but still got perfect results,

I also tried with the input-xml on as you described, but didn't see any difference.
I couldn't see any link problems either - on that example anyway.

Hm, actually, that server has the tidy extension available... hm, no. I downgraded it to the commandline executable also, and still got proper --with-&lt;target&gt;
results in my preview pane.

I can't replicate on the servers I have here, sorry.

sspr’s picture

sspr commented

Some crucial information I forgot to mention is that I was using the 4.7.0 version of import_html. I'll first test with the HEAD version before proceeding... anyway thanks for testing all this !

sspr’s picture

sspr commented

Using HEAD, it only got stranger. I've raised the debug level to '3': and got the different stages of processing. The last 'textbox' displays the content below:

XML DOM being scanned for XPATH data extraction

<head><title>The OROCOS Real-Time Toolkit Installation Guide</title>
		<meta name="description" id="teaser" xml:id="teaser" value=""/><meta name="generator" content="DocBook XSL Stylesheets V1.68.1"/><meta name="description" content="
	This document explains how the
	Real-Time Toolkit of Orocos,
	the Open RObot COntrol Software project
	must be installed and configured.
      "/>
		</head>
		
		<body><h1 id="pagetitle" xml:id="pagetitle">The OROCOS Real-Time Toolkit Installation Guide</h1><div id="content" xml:id="content">
									
					<!--Couldn't selectively extract content, Imported Full Body :(-->
					
					<div class="article" lang="en" xml:lang="en"><div class="titlepage"><div><div/><div><h3 class="subtitle"><i><span class="emphasis"><em>Real-Time Toolkit Version  ........ rest of file looks good.

Next, it says:
lookup for xml:ID content failed, doing query instead

and the Got Body is plain empty:

Finally, this is the 'official' output:


    * Importing 1 files now
    * We already have 'rtt/doc/orocos-installation' in the system as 'node/57'. Overwriting it with the new import
    * Fetching content from '/opt/www/bruyninckxh/web/www.kindyn.org/stable/rtt/v1.0.0/doc/orocos-installation.html' now.
    * Failed to find a body, anything with id='content' in this page
    * Processed page to extract content. Title:'The OROCOS Real-Time Toolkit Installation Guide'
    * Import of /orocos-installation.html did not quite validate. New page is NOT being added.

I had to set the new html2simplehtml.xsl stylesheet because I upgraded from 4.7.0.

Also the ../configure --with-<target> is still wrong, but I guess that is PHP/XML version specific...

sspr’s picture

sspr commented

It seems the following piece of code is not working well in PHP4:

function xml_query($xmlnode,$query){
  if(extension_loaded('xsl')){ // PHP5
    $xp = new DomXPath(xml_getOwnerDocument($xmlnode));
    return $xp->query($query, $xmlnode);
  }
  // PHP4
  // TODO test!
  $xmldoc = xml_getOwnerDocument($xmlnode);
  $xp = $xmldoc->xpath_new_context();
  return $xp->xpath_eval($query, $xmlnode);
}

It returns for my document two elements: {1, Array} if I debug() the results.

Maybe this page can help,

sspr’s picture

sspr commented

Upgrading the site to PHP 4.4.4 solves this issue with the module untouched. It is hence a PHP 4.3 bug. Maybe you can note on the module page that it is PHP 4.4+ and warn for 'reduced' functionality in lesser PHP versions.

dman’s picture

dman commented

Thanks for the follow-up.
I'm adjusting the help document now.

dman’s picture

dman commented
Component: Code » Documentation
Status: Active » Fixed

I've Noted this in the docs now
I don't think it's worth working around the issue for the old broken PHP XML, just leaving it as a known issue.
Thanks for the support.

Anonymous’s picture

(not verified) commented
Status: Fixed » Closed (fixed)