Announce: First Release of Import HTML Module

As briefly discussed in passing a month or two ago, I've been intending to do this properly.

By the time I got CVS access last year I'd lost interest a bit, but due to a few requests I've re-activated this project and released a very hairy first go at the Import HTML module

I'll copy the project overview below (sorry it's big) - please take note of the requirements.

I'd be receptive to any input, but it's deployment testing and major bug squashing at the moment - no 'feature requests' :-B I've got too many settings already.

---

Drupal Module: Import_HTML

Synopsis

Facility to import an existing, static HTML site structure into Drupal
Nodes.

This is done by allowing an admin to define a source directory of an
traditional HTML website, and importing (as much as possible) the content and
structure into a drupal site.

Files will be absorbed completely, and their existing cross-links should
be maintained, whilst the standard headers, chrome and navigation blocks
should be stripped and replaced with Drupal equvalents. Old structure will be
inferred and imported from the old folder heirachy.

• Requirements
• Usage Detailed step-by-step
• Intent What it's indended to to
  • Methodology Exactly how it does it, at a
    coder level
  • Notes Issues arising, and some detailed
    explanations
• Guide Reference section
  • Setup Requirements, and Installing for the
    first time
  • Import Templates XSL. With great power
    comes great complexity
  • Settings Explanation of the user
    settings
• Development / TODO

Requirements

Before you begin

See the setup section for details. Because of the
number of settings, this is not just a point-and-go module.

• XML/XSLT support on the server. Check your php_info().
• HTMLTidy - Either with the PHP module or the commandline version.
• Some understanding of XSL for advanced template translation.
• Some libraries of my own (bundled) to actually do the XSLT

Usage

This module uses no database tables of its own. It requires XML support on
the server, this can be tricky if it's not already enabled.

Given a working system, the process is thus:

1. Visit the admin/settings/import_html page and
   check the settings.
2. If all values look OK for now, you can try a test run by visiting
   admin/import_html/demo . Choose a 'page' sort of page, not a portal or
   layout-rich sort of thing. The demo will scrape the given file and import
   it to the system. Some of the new navigation features will not be apparent
   yet, as they apply only to large-scale imports, or at least imports that
   have a defined siteroot.
3. Try opening the 'admin/import_html' main page and defining
   a source folder. Enter the root path of the site you wish to import and
   continue. The UI should display a treeview of the
   files you can selectively select for import.
   
   It's recommended to just try one page at a time to begin with.
4. Upon importing a page, a new node should be created. The object of the
   import templates is to trim down the content block to its unique
   value. This will probably require some template tuning, so make a
   new template (copy the existing html2drupal.xsl), select it (enter the new
   name in the admin page) "http://www.dpawson.co.uk/xsl/sect2/sect21.html">tweak the XSL and try
   again.
   
   If you are extremely lucky, or don't care too much about the extras, you
   can go straight to bulk import.
5. If you need to check how the the images are turning up, they can safely
   be imported as well using the previous interface. They will be copied,
   structured in the same folders they were in originally, into the directory
   configured in the admin/setting. Imported pages will have their links
   rewritten to find them there.
   
   Two type of content are being imported, depending on file suffix. 'Pages'
   (html) - which become nodes ... and eveything else, which becomes 'files'.
   Already it seems that file suffixes are not good enough,
   should the suffix list be editable, or should I scan the files
   themselves?
6. When you are happy that the body field is as tidy as it's going to get
   (test several pages), you can try a bulk import. This may fill up your node
   collection a bit, so be prepared to delete them if things don't work
   perfectly first time. Many static sites have whole sections that are not
   structured the same as the rest of the pages.
7. On input, a menu structure and a bunch of aliases will be
   auto-generated. These can be manually adjusted easily. For instance, the
   menu branches will initially be named after the document titles found in
   the directory structure. Which is great if you used a decent folder
   heirachy, but some of the labels can probably be tidied up a bit. For that
   matter, after input, you can safely re-arrange the menu structure
   altogether, shifting whole sections to different places without worrying
   about links breaking. These changes will show through in the menu, sitemap
   and breadcrumbs but not in the pathalias. There
   may be issues navigating to pages deep in a menu where the parent has not
   been imported or created yet. This is normal Drupal behaviour when making
   menu links to non-existant paths.

By following these instructions, you should probably be able to end up
with a version of the old content in the new layout. For large sites (200+
pages) some extra tuning may be neccessary, eg using different templates for
different sources.

Incremental imports, processing just sections at a time, or repeated
imports as you tune the content or the transformation should be
non-destructive. Re-importing the same file will retain the same node ID
path, and any Drupal-specific additions made so far.

Intent / Theory

This is intended as a run-once sort of tool, that, once tuned right on a
handful of pages, can churn through a large number of reasonably structured,
reasonably formatted pages doing a lot of the boring copy& paste that
would otherwise be required.

The existing file paths of the source content will be used to create an
automatic menu, and therefore a heirachical structure identical to the source
URLs. With module:path, appropriate aliases will also be created such that
this will enable a drupal instance to TRANSPARENTLY REPLACE an existing
static site without breaking any bookmarks!

Methodology Overview /
Tasks

A peek under the hood into what happens in what order

• Facility for spidering/enumerating existing source files. (the
  admin/import_html page)
• Define import rules - choose an XSL stylesheet, set some parameters on
  it, configure presets for the imported
  pages.
• Expose selective selection of files to import (admin UI)
  [screenshot]
• Import each source file by way of sequential
  1. (Optional) download/copy of files to local
     mirror site.
  2. Processing with html-tidy, to prepare for XSL transforms
  3. URL-rewriting via XSL. All hrefs are redirected to the new
     pseudo-location aliases, all srcs are redirected to somewhere under
     /files.
  4. Content-scraping via XSL (XSL stylesheet will probably have to be
     customized to each source site)
  5. Or content-scraping via RegExps and heuristics
  6. Deduction (as much as possible) of meta-information like page
     title, author,date
  7. Validate nodes and save them with node-insert calls
  8. Extra API-insert calls to create menu navigation and path aliases
     (taxonomy leverage also?) Two types of alias: /path/filename
     and /path/filename.html can be created for now.
• Pages are now first-class nodes, and can be administered through the
  CMS as usual.

Notes

The more valid and more homogenous the source site is, the better. A
creation using strict XHTML and useful, semantic tags like #title #content or
something could be imported swiftly. One with a variety of table structures
may not...

Of course, this tool is supposed to be useful when dealing with messy,
non-homogenous legacy sites that need a makeover. 'todo'>Sometimes regular expression parsing may come to the rescue for
content extraction, but that's not implimented yet.

I'm choosing XSL because I know it, it's powerful for converting content
out of (well-structured) HTML, and I've had success with this approach in the
past. Others may object to this abstract technology (XSL is NOT an easy
learning curve) but the alternative options include RegExp wierdness or cut
and paste. (which I may patch on as alternative methods -
or someone else can have a go) Both approaches I've also used
successfully in bulk site templating (over THOUSANDS of pages) but it's my
call. Making your own XSL import template is
non-trivial.

Guide

Installation/setup

XML Support

The module can use either the PHP4 and PHP5 implimentations (which are
quite different) but the PHP modules do have to be enabled somehow.
This can be tricky as they often require extra libraries to be put in your
path somewhere. Please don't ask me for instructions, every time I've done it
it hurts my head.

HTMLTidy Setup

The module also uses the
famous HTMLTidy tool. There is now a PHP module that
impliments HTMLTidy natively, but that needs to be installed and enabled.
If you don't have access to that, we can run it from the command line. Find
the appropriate binary
release of HTMLTidy for your system, and place it in your PATH, in the
modules install directory, or wherever you like, then define the path to
the executable in the settings.
This works fine under Windows too.

Import Templates

An import template defines the mapping between existing HTML content and
our node values. It uses the XSL language because of the power it has to
select bits of a structured document, for example
select=\"//*[@id='content']\" ... will find the block anywhere
in the page, of any type with the id 'content', and
select=\"//table[@class='main']//td[position(3)]\" Will locate
the third TD block in the table called 'main'. Both these examples would be
common when trying to extract the actual text from a legacy
site.

You can begin with the example XSL template, this contains code that
attempts to translate a page containing the usual HTML structures like
(either title or h1) and (either the div called 'content' or the entire body
tag) into an import structure similar to that used by the
XML import module. This structure may change but for now it should
looks something like: [example]

It's likely that whatever site you are importing will NOT be shaped
exactly like we need it to translate straight using this format. You have to
identify the parts of your existing pages that can reliably be scanned for to
define content, then come up with an 'http://www.w3.org/TR/xpath'>XPath expression to represent this.

If your source, for example, didn't use nice H1 tage to denote the page
title, but instead always looked like

<font size='+2'><B>my
  page</B></font>

... your template could be made to find
it, wherever it was in the page select=\"//font[@size='+2']/B\"
And proceed to use that as the node title.

No, the code is not pretty, and if Regular Expressions are a foreign
language to you, This is worse.

But this is why developers have been ranting for the last ten years
about using semantic markup!!

The uniformity, and the usefulness of the metadata detected in the source
files will play a big part here.

It's easier to develop and test the XSLT using a third-party tool, I
recommend Cooktop. Be sure to set
the XSL engine to 'Sablotron' which is the one that PHP uses under the
hood.

Although it would be possible to configure a
logical mapping system to select different import templates based on
different content, at this stage the administrator is expected to be doing a
bit of hand-tweaking, and predicting all possible inputs is
impossible. Some of this sort of logic can
however be built into the powerful XSL template, if you are good at
XSL

Once importing is taking place, you can even filter it more to improve the
structure of the input, for example by removing all redundant FONT tags, or
by ensuring that every H1,2,3 tag has an associated #ID for anchoring. Yay
XSL.

Settings

On the admin/settings/import_html screen, you can (if you wish):

• Choose the import template. These templates
  translate between the existing page structure and the raw content blocks.
  Currently you enter the name of the xsl file directly.
  If I make this a select, some flexibility is lost
• Customize a parameter used in the import template - the id of the real
  'content' block of the source documents. This could be
  extended into a wizard to work towards an all-purpose template, but that
  will probably never happen. Can't predict how broken the import sites may
  be.
• When a site is imported, it must bring along some of its baggage.
  Images and suchlike. You can choose where they will end up.
• When the imported site is given new URLs (reflecting the original path)
  we can publish the new nodes in a 'subsite' by applying a prefix directory
  to the aliases they are issued. The existing old links will be written to
  point to where the imported neighbouring pages are EXPECTED to end up.
  Incremental processing means they may not always be there until the whole
  site is done. Link checking (preferably on-the-fly) would be a nice tidy-up
  process.
• The new URLs generated for the new pages are url_aliases based on the
  original paths. You can choose to have tidy (no suffix) or legacy (old .htm
  or whatever suffix) aliases - or both.
• As the input is (a fragment of) Pure HTML, the content filter (input
  format) must be set correctly. I choose to define a blank filter, which
  doesn't even add extra BRs, but you can override that if you wish.
  Should this option be hidden, or is it
  useful?
• If using non-native HTMLTidy support, the path to the tidy executable
  should be defined. Security issue giving commandline
  access to a bin? Should move into server-side settings?

Notes on the Treeview Interface

Files and folders beginning with _ or . are nominally 'hidden' so are
skipped and do not show up on this listing. While it's
possible to list a thousand or so files, It may be a good idea to allow the
listing to be more selective, to scale to larger sites.

Development / TODO

As mentioned in Usage, this module uses no database tables of its own.
Pages are read straight into 'page' nodes. I guess it could feed into
flexinode if your import files had extra parsable content blocks, and I've
sucessfully used it to import other random XML formats (RecipeML) although
the advantages of doing so are limited.

It's easy to imagine this sytem set up as a synchroniser, that could
re-fetch and refresh local nodes when remote content changes. This would
involve recording exactly what the source URL was (which isn't currently
done) but would be a fun feature.

I may fork off the page-parsing into a pluggable method, so that a regexp
version can be developed alongside, and be used for folk without XSL
support.

How to leverage this to import a local site to a remote server? You must
either unpack the source files somewhere on that machine, then provide the
absolute path where the server can find them, or upload a
zip package and I'll try to take it from there.(TODO)

Also TODO is a 'Spidering' method to try to import URL sites.
Way in the future!

TODO Allow settings to set import content type to something
other than 'page'

TODO Find a way to map more meta-data from the original page
(assuming there is any to be extracted) to Drupal properties, eg get the
contents of META keywords into Taxonomy associations

TODO There are issure when a page links directly to a file
that would be regarded as a resource via an href. Most hrefs are re-written
to point to the new node, but things like large images or word docs get
imported under 'files'. The XSL rewrite_href_and_src.xsl attempts to correct
for this, but there may be some side-effects. Always run a link checker after
import.

---

Dan