remote access vs local import?

Thanks for the excellent thought-provoking (open-ended) question (and that excellent article).

I know there are obvious pros/cons for both approaches (import vs. external data), which you might already know... and I realize this is an open topic and not a direct question... so I will just start out by explaining why I made some choices =)

First of all: we do have a tiny bit of external data-on-demand, the holdings information table for each record.

Now, I chose to import for different reasons:
* Simplicity. Stuff is nodes (How simple is that?) =) The module was also meant to be gimme-OPAC-URL-you're-done simple... admitted, I didn't think too much about architecture back then =)

* Extensibility. Again, they're nodes. How could you use flags, comments, etc. if they're not? I think SOPAC and others have this basic problem and I think have heard some complain about that.

* Speed. Yes, I know the import is kind of dog-slow (currently). However, overall, we've had good experience with a previous version (5x slower!) version of millennium.module's harvester crawling portions of our OPAC (35k+ records) for one of our subject gateways (just called cron every 5 minutes for a day or two... slow but super-simple). For visitors, you can quickly put in things like Boost or other solutions to make the site totally scalable.

Of course I missed the mark on not making it CCK compatible long ago. And yes, Feeds is excellent for doing part of millennium.module's job as you mention. I plan to look into Feeds soon.

I've not yet pondered too deeply on fetching external information; we're just beginning to do so at #358765: Integrate with other external data sources (WorldCat, LibraryThing, Open Library). The remote data issue is not a simple one.

I don't think on-demand field population is the way to go, however neither is importing with traditional cron jobs bootstrapping Drupal. Some sort of pipelined data harvester that runs separate from Drupal (or using minimal bootstrapping) could/should take care of this. Supporting this, I've also seen some approaches to RDF mashup presentations where the recommendation is to copy portions or complete datasets locally...

Re-reading that article, I think the module could take a hybrid approach:
1) I don't think each biblio field (e.g. ISBN, Notes, etc) should actually be in CCK, but rather in their own table(s). Then we'd have Views integration to show them however we like. This is approach (3) mentioned.

2) We are also using the "lazy-bridging" (#2) approach for item holdings information: the node id can be dereferenced into an OPAC URL and bib record number, from where we load the holdings info via HTTP and then show it via AJAX.

3) For aggregating additional information from different providers into the node would again be something like approach (3)... a harvester would fetch and locally store/update metadata that would get added into the node at display time, or with views integration. I am yet unsure if MYSQL is a good place to store this information (just using it as a sort of filesystem, using the node id as the key) or if a nosql-type solution like CouchDB (which can search through schema-less data) would be a better place. I'm leaning towards MySQL-as-a-filesystem right now to keep it simple and don't have any use cases for querying the harvested data using the data itself.

As I see it, "database slurp" into local nodes has a couple of benefits:

• The possibility to reuse all the node-based features available in the Drupal ecosystem,
• specifically, the ability to create a local faceted search index using Apache Solr, and
• independence from the lack of proper live API:s on the ILS side,

Having read the article it occured to me that maybe Millennium Integration could also optionally support a Method 3-type model without generating the nodes in advance. A local database slurp into the database (like now), but the records would be accessed via a record identifier uri and the nodes would be created lazily.

Because bibliographic library data is pretty static overall, I wouldn't worry about the latency issues too much. And apparently Drupal does support a very large amount of nodes. (But if that's a problem, maybe the lazy-node approach could help.)

Anyway, I guess this approach isn't suitable for every situation, but it does have some clear benefits.

I see. By "lazy-node" do you mean something like:

a) There are no nodes on the Drupal site yet.
b) A user tries to access something like http://example.com/millennium/record/b123456x
c) Millennium module gets the metadata for b123456x
d) Millennium creates a node for b123456x for the current OPAC and redirects user to the node
(say http://example.com/node/789)

OR:

just a,b and c PLUS showing the record metadata to the user (no node creation, hence no taxonomy mapping, no metadata storage, etc... something like the Millennium Filter submodule?)

The first one was kind of what I meant. Larry Garfield's article says "since you don't want to allow editing of that content through Drupal there's really no need to make nodes out of them anyway" regarding method 3. But there might be other reasons you would want nodes: enabling commenting, CCK-fields, and maybe to use other 3rd party modules which depend on nodes. Maybe, instead of creating a node each time, a node could be created if a user wants to write a comment.

But I'm no expert on this, and I'm starting to think this might not be worth it, since the use cases are beginning to sound a bit theoretical. It wasn't something I myself would actually need at the moment. This was more of a vague idea: "hey, maybe you could separate node creation from metadata storage".