Closed (fixed)
Project:
Import HTML
Version:
6.x-1.0
Component:
Code
Priority:
Normal
Category:
Support request
Assigned:
Unassigned
Reporter:
Created:
20 Nov 2009 at 16:25 UTC
Updated:
30 Nov 2009 at 04:09 UTC
Hi,
I'm currently trying to run a "bunch" of ASP pages (Microsoft "Active Server Pages") through the 'import_html' module, so far without any luck. The site is organized into a deeply nested hierarchical directory structure, where most paths end with an index.asp file, so the directory it is residing in could make a fine node title in Drupal.
However, even with the simplest/smallest files I'm getting errors like:
Had XML trouble with the document 'sites/default/files/legacy/index.asp'
Import_HTML failed to initialize or parse XMLdoc input
Failed to process a node out of file 'index.asp'
After some tests I think, 'import_html' currently can not process embedded statements like <%= weekdayname(weekday(date)) %>, <%= date() %>.
So basically my question is, if 'import_html' theoretically could handle stuff like this or if something (e.g. the syntactical similarity with PHP) makes this impossible. Is this a matter of a "customized" new XSL template, or would processing ASP files require a completely different processing logic (stripping all ASP code would be fine with me, so maybe an additional "Remove all old code" similar to "Remove old javascripts" would suffice?).
Also I ran into some other difficulties caused by some bogus design decisions made by the website-to-be-imported (e.g. dots (".") in directory/ pathnames, upper casing in file/directory/pathnames etc.); those things would have to be cleaned up during - or before - importing the site into Drupal. So the second question would be, if such stuff falls into the intended scope of the 'import_html' module, or if it would be required to get this fixed before approaching the import process itself (I'm not a programmer, but recursively changing file and pathnames and sychronizing those changes inside the files seems not quite trivial; please correct me if I'm wrong, that would get my hopes up ;)
Thanks for this great module & greetings, -asb
Comments
Comment #1
asb commentedSmall addition: Watchdog logs the following error(s) multiple times:
The
invalid element name in Entityrefer to different lines, probably in the source document (?)Comment #2
dman commentedSound like "fun".
But yes, this sort of thing is what it's designed for.
With preprocessor languages like asp, php, (cf, jsp) you can't import the source, you'd got to import the result. The actual HTML as the browser would see it.
Among other things, you've got to let the system build the page and put content in it first!
So what you need to do is take a static mirror of your own site, ant run import_html on that.
There are instructions for that in walkthrough.htm - "Prepare the content".
A 'remove all code' pre-filter could be possible (can't do it with XML) BUT it would not get the result anyone wants.
Fixing pathnames is within the scope - but only up to a point. Relative links, Mixed caps should be fine. Case-insensitive rewrites? um, not yet. I respect case, so if the input is confused, it will remain so. Could be added as another little process if needed.
Dots in folder pathnames - um, maybe? That's curious, but not illegal. Not impossible, I imagine the name-parser may want to chop it off, when guessing the menu title, but the import should still work.
Comment #3
asb commentedHi Dan,
many thanks four you helpful answer. I've continued my analysis of the site I have to process and would like to give some feedback about the import/migration progress.
> With preprocessor languages like asp, php, (cf, jsp) you can't import the source, you'd got to import the result.
> The actual HTML as the browser would see it.
>Among other things, you've got to let the system build the page and put content in it first!
I think I have some kind of a special case here; the ASP code doesn't build the pages (as PHP does for Drupal), it simply adds a few things to mostly stand-alone "HTML" pages (like includes), as it seems it's nothing essential that needs to be preserved. Thus I preprocessed the pages with some Regex logic and stripped all code fragments (stuff embedded into
<%= ...). BTW, this is a mid-sized site with a total of 6.5 GB of data (~135.000 files, ~11.200 directories).I had to run several tests until I realized that this is mostly content other than html pages (images etc.) and some weird "data" records (
_vti_cnfin hidden directories). 'import_html' processed those folders anyway; I think, the "File Exclusion Pattern" do - as the name indicates - indeed applies for files only (it'd be a nice addition if this would take care of directories, also!).However, bogus imports of this kind can be easily resolved easily with the Views Bulk Operations (VBO) module, looking inside the imported node bodies for strings like
_vtian then by deleting the bogus nodes. For the next runs I removed those data records before importing manually withrm -rf `find . -type d -name _vti_cnf`In the end I got approximately 3,000 pages to import, with a total of about 70.000 additional media assets. I could split the site into three main sections resulting in three huge import runs; splitting one level deeper would have resulted in approximately 90 (!) import runs, so I decided to go for the simple approach. Importing even these quite moderate amounts of data causes some load on the server; each run took finally between 1 and 6 hours. The server - a dual core machine - had to take a continuous load of approximately 3 to 4. To get there, I had to increase several system limits:
Both server-side limits are well indicated by Drupal error reporting; e.g. a too small max_packet_size outputs:
Too small PHP memory limits are lindicated like this:
These helpful error loggings are an real improvement compared to D4/D5 where PHP memory related issues often simply lead to a non-indicative WSOD. It's very nice this way, at least as long you have a server with some spare horsepower at hand ;)
> Case-insensitive rewrites? um, not yet. I respect case, so if the input is confused, it will remain so.
> Could be added as another little process if needed.
I tried preprocessing the source files with a small script (LowerCase-R by Felix Stahlberg) but didn't succeed fully; renaming the files and directories is trivial, but fixing all references inside the files (looking for
href,src, etc.) didn't get all files; I think this would need to be handled smarter, e.g. with a XML magic;rewrite_href_and_src.xslseems to be the right place for this. Anyway, I went the shortest path and left the mixed upper/lower case directory and filenames alone; the positive side of this is, that the old URLs will be preserved ;)Anyway, if you have time to look into case-insensitive rewrites: I found two XSL transformations:
I understand much too litte about XML to make use of stuff like this ;-/
> Fixing pathnames is within the scope - but only up to a point. Relative links, Mixed caps should be fine.
> Dots in folder pathnames - um, maybe? That's curious, but not illegal.
> Not impossible, I imagine the name-parser may want to chop it off, when guessing the menu title, but the import should still work.
Dots in folder pathnames were processed cleanly; I remember having heavy difficulties with such stuff in the D5 version of 'import_html', but the D6 rewrite appears to behave significantly more robust.
Other relevant import parameters:
With these settings I got a pretty good "mirror" of the old site's URL and content structure.
Up to this point the import process went quite smoothly - but finally I ran into an issue I couldn't solve yet: UTF8 handling seems still to be quite broken in 'import_html'. The import files were cleanly encoded in UTF8, in the imported Drupal pages I'm getting lots of (double?) encodings like those:
This is really bad since I have been unable to process these characters/encondings with the Search & Replace Scanner module (it locates the strings, but seems not to process them - maybe it also has problems with UTF8 ;) Do you have any idea what is going wrong here?
Anyway, so far I'm quite impressed with the D6 'import_html' module. Thank you for this great module!
Greetings, -asb
Comment #4
dman commentedWell that sounds like a whole heap of fun! Thanks for the feedback.
Removing
<% %>tags could be done - but not by XSL, so a search & replace is the thing to do.Yeah, at some point the machine needs to be beefed out a little when handling tens of thousands of items. I think that the biggest weakness right now is the way it uses the batch API. This is the one that loads too much into memory at once, producing max_allowed_packet and other blowouts.
To really do justice to a site that size, I'd have to move the processing into cron or something.
But, as you can imagine, the more I break things down like that, the slower it gets. Several hours of runtime is pretty heavy. I wonder how much of that was just file copy.
Note that LOGGING itself becomes expensive quite quickly. A significant amount of processing and space is taken by providing debug feedback. It's a tricky balance for me.
Yeah, it looks like the file exclusion path could be tuned a little. I was intending for it to be able to skip the ASP _vti files, but I guess that didn't get tested.
XML to change filename case is possible, but as magic as it is, I'll admit this is not a great thing to do with XML. This is done MUCH better as a code process, pre or post transformation. I have a valid DOM, we can do useful things to all (@src|@href) attributes. Stick it on the wishlist.
Aaaand. UTF8
Yeah, I have trouble with character encodings. Part of it never got solved because PHP4 XML handling was so awful I think I built in some work-around just to get it to validate. There may be a solution, but I don't know it.
Early in the process, I ask html-tidy to convert strange characters to numeric entities. This seemed to be required before the XML process would even touch it.
Then, every time a process is run, those entities came out into characters again, (gah!) so I had to re-encode.
And at the end, although I was getting valid HTML results with encoded entities (your results may be different) - those numeric characters look pretty ugly in an editor - unless you are using a WYSIWYG.
Simply, if I tried to leave non-ASCII characters in the input during processing, errors happened everywhere.
All I can imagine is to replace a number of safe characters into a visible form again at the end.
... or just see if PHP5 character-handling can or should handle this now.
Ah well, I'm glad the process is helping you with at least some of your job today. It sounds like there are some things that could be done to speed it up though :-B
Comment #5
asb commentedHi Dan,
IMHO the batch API is an incredible enhancement to 'import_html'. But yes, maybe it could be fed with smaller chunks to process at once. I'll have a look in the VanDyk book, maybe I'll get an idea. If everything else fails, I'll have to spend a couple of hours to manually trigger smaller chunks of import processes.
Regarding the file exclusion path: Microsoft Frontpage (and probably Expression Web, too) seem to create a structure like this:
If 'import_html' could exclude directories and it's contents starting with an underscore, the bogus stuff would be skipped. Actually, the recursive delete operation mentioned above does the trick well enough, so don't bother too much about this.
Currently, the real problem for me are the UTF-8 encodings:
<title>Bücher</title>) and - very strange - in the heading (e.g.<h1 class="title">Bücher</h1>) are numeric (mis-/double) encoded. (note to self: figure out why S&R scanner doesn't work!)http://mysite.tld/bücher/cover.jpg) becomes inaccessible after import, at least in my case (source files came from Windows, which should have UTF-8 encoding for files, and were processed under Debian "Lenny", which should be UTF-8 clean, also; on the bash, "bücher" becomes "b_cher", if I rename this to "bücher", the webserver still doesn't find the files). According to the Transliteration project page, "PHP 5 doesn't fully support Unicode characters in filenames, and may not be able to access those files"; maybe that's the problem here? It would be nasty, but umlauts in directory names could be taken care of before the import if absolutely necessary (however, that raises the problem of fixing references to renamed files again ;)As far as I see for now, these broken encodings are a real weakness of the 'import_html' module.
For path- and filenames: Would you consider integrating the Transliteration module for imported files and transliteration for URLs through Pathauto module (which now also uses the 'transliteration' module [1]: "Module developers who want to make use of transliteration to clean input strings may use code similar to this [...]")?
Since this was initially a support request about existing features of the module, I'm now closing it. If you have spare time and are interested in working on the mentioned problems, maybe we can at least partially sponsor some enhancements - and there's still one more import to do ;)
Thanks & greetings from Berlin, -asb
Comment #6
dman commentedYep. "_*" folders should be getting skipped. They just aren't. Was/is a limitation of file_scan_directory - or at least my use of it. Probably a small fix.
Wacky characters in unexpected places is just something I've not had to test for (or pretended they didn't exist).
Adding in an unescape in known places like node->title makes sense though, so that's do-able.
I get the feeling there is a pure-XML way of letting them through however, but it was beyond me last 3 times I tried/
In pathnames? o_O ... Gee I dunno.
I don't THINK I'm messing with that... Oh yeah, I am. There's something that flattens URLs into 'safe' strings. A handmade version of transliterate (predates it by ages). You are right, looking at incorporating transliteration.module properly may be the way.
That needs revisiting I guess.