Add Spider capability

Hm.
Looking at the current code I see no actual spider support there yet. Looks like it can get one page on request, and extract simple information out of it.
I'm already doing that much - with a whole bunch of extra edge-case support that arose from testing.

I'm looking at an eventual way of live-mirroring a site - to import it. I've either gotta use batch jobs, cron daemons and all sorts of status semaphores to do it right. ... or do it realtime on-demand like a caching proxy ... which is a lot more lightweight, and is almost working for me.

I'm sure there are other spider libraries out there that will help, but this one (dataminerapi) isn't it. Currently the most robust way forwards is still wget (maybe it could be daemonized?) and then import via batch.

I'm currently exploring various spiders for our site migration. Wget is one. Others I'm testing include: Pavuk, HTTRack

No idea if the technology is compatible, but when I researched this, I found that the open source Scrapy (www.scrapy.org) had a lot going for it.

Probably not. scrapy being python, I can't see my PHP talking to its API (if it has one) without hitting the command-line.
Currently I can retrieve files OK, and I can parse out links OK using the DOM. All that it would take to turn that into a spider is to loop that process. Adding python to the mix would not be much fun.

FWIW, I do have a background wrapper/scraper script in development that fetches/imports foreign pages on-the-fly as needed.
But a full spider thing needs a full daemon process - bigger that a PHP 'batch' script, so it's not going to happen entirely within import_html.module.
The best option is still : to the spidering yourself to a local flat-file mirror, then point import_html at it.