An option to go through all the bibliographic records already in the database and refresh the biblio_data arrays as well as taxonomy mappings from the marc record already stored in the database.

If you change the settings for the taxonomy mappings or change the code which maps the marc fields to the biblio_data array, you will want to quickly refresh the actual data to reflect the new definitions.

CommentFileSizeAuthor
#4 millennium-629778-3.patch5.33 KBtituomin
#2 millennium-629778-2.patch5.3 KBtituomin
#1 millennium-629778.patch5.16 KBtituomin

Comments

tituomin’s picture

tituomin commented
StatusFileSize
new millennium-629778.patch5.16 KB

This patch does the following. It adds a checkbox to the Millennium import form, where the user can choose that s/he wants an offline update of the existing records. Then, there are a few modifications to the batch import functionality reflecting this option: if the checkbox is checked, we will bypass the mass fetching from the opac and get the marcs from the database instead.

tituomin’s picture

tituomin commented
StatusFileSize
new millennium-629778-2.patch5.3 KB

A better version, just a small database optimization: instead of querying for the bib data repeatedly in a loop, we query once and fetch the results in a loop.

janusman’s picture

janusman commented
Status: Needs review » Needs work

Great! Just a small thing...

@@ -653,10 +678,35 @@ function millennium_batch_import_process
[...]
+      // Note: bypassing string escaping on purpose.
+      "SELECT bib_recnum, biblio_data FROM {millennium_node_bib} WHERE bib_recnum in ($chunk)");
[...]

Could you please use db_placeholders()? ... see http://api.drupal.org/api/function/db_placeholders/6 for docs and examples =)

tituomin’s picture

tituomin commented
StatusFileSize
new millennium-629778-3.patch5.33 KB

Right, always learning! Here goes.

BTW, the "offline" refreshing is taking surprisingly long, I might have to check if there still is some data which is fetched from the opac.

tituomin’s picture

tituomin commented

Yes, it seems that millennium_marc_to_nodeobject calls millennium_get_holdings_info to get the lcc field. I could create another function parameter to disable the online fetching of holdings info. But that would erase the lcc field. So we should save it from the biblio-array and reset the field to the old value inside millennium_marc_to_nodeobject.

The holdings-fetching operation could also be done elsewhere, since the function name marc_to_nodeobject really doesn't seem like there will be online data retrieval and scraping going on...

Another thing we should test is if the item-recnums will stay intact during this process. (It currently is bib-recnum based).

janusman’s picture

janusman commented

I think not only that, but it currently also tries to fetch information from the library of congress (tables of contents, publisher description, etc) if it finds certain links on 856 fields.

Maybe this kind of enrichment from outside sources should be a global switch (maybe even separated into their own module) as it seems kind of hackish to just add on this kind of info (I'm thinking this module could let the admin choose other sources to enrich records from, like OpenLibrary, LibraryThing, etc). I'll ponder on it.

For now, yes, you have to disable it somehow in order to make refreshing the nodes quicker... Maybe by adding two checkboxes on the main configuration form: one to enable getting the LCC field from the items if the marc doesn't have it, and the other for external calls to LOC, etc).

janusman’s picture

janusman commented
Status: Needs work » Needs review

I just committed a change that moves LOC fetching into it's own module, and disabled LCC fetching from the holdings when none is in the MARC. This should improve things.

Will test this soon.

janusman’s picture

janusman commented
Status: Needs review » Fixed

committed!

janusman’s picture

janusman commented

Note: if the "Enrichment" submodule is active and its options are enabled in the Settings > "Enrichment" tab, then refreshing will not be completely "offline" in that it will still try to fetch external data (when applicable; for instance the LOC enrichment needs certain 856 fields present to trigger it).

tituomin’s picture

tituomin commented

Thanks for this! Now the offline update is really fast!

tituomin’s picture

tituomin commented
Priority: Normal » Minor
Status: Fixed » Needs work

Oh, it was just the cache for the get_holdings_info. add_availability_to_node still gets called every time. We should continue working with the global "offline" switch to disable this if the user wants.

janusman’s picture

janusman commented

A possible problem:
* Right now we are "recreating" the biblio and node data and not "refreshing" (which to me would mean something like "change only things that have changed").

If we mean "recreate", and we offer the option to do it "offline" then data that was already on the nodes will be lost.
If we mean "refresh" (change only what has changed) we can only do it if we check the online data to see if it has changed (or, at least, compare to a locally-cached version of it if we are OK with a little staleness.)

So, which are should we aim for?

tituomin’s picture

tituomin commented

You're right, I think "recreate" probably fits what I had in mind much better.

To me this feature is more immediately useful as an aid for development and debugging (after having changed some of the MARC-related code).Of course it *should* also be useful in real life maintenance scenarios (for example: "we have these 100 000 records on a production site, and we want to quickly add one new taxonomy mapping for each one"). But this was never meant to be a way to actually see if the data has changed in the Millennium system.

But it is a real problem if we lose some data in the process.

I think the recreating should only apply to the data that is based on MARC records. In general, a clear distinction should be made between

  1. data that is relatively stable (static bibliographic MARC metadata is note updated very often, at least in most libraries, I think)
  2. data that changes more quickly (item data, locations of items)
  3. "live" data that changes very quickly (availability)

This "recreating" should only apply to #1. And we should ensure that #2 and #3 won't get erased. I actually think it's a good idea to only cache #2 and #3 type data locally, and not to save them in the same way that we save #1. But of course, that would disable using location data as facets, that's the whole point of add_availability_to_node, right?

So my goal is to make this feature refresh only the MARC-based stable data and leave everything else intact. That might be a bit difficult, I'll have to see.

janusman’s picture

janusman commented

I'll add more questions than answers to the discussion =)

I brought this up mainly because of the new "enrichment" sub-module...

Cover images, links to worldcat, tables of contents from LOC and other stuff that sub-module is fetching might be part of (1) that you mention... data that doesn't change much.

However, I had considered the enrichment sub-module (or your own similar development) would just add data to the same biblio_data array at node import time; however it might be wise to instead keep it separate (store it in a different place) *when possible/desirable* and then just combine it at hook_nodeapi($op = "load") time (or some other event) so that keeping metadata from different sources separate also allows us for separate harvesting/updating/maintenance.

Now, how to decide if data is to be kept separate or not, *might* depend on:
* do we want taxonomy representing that data?
* do we want to index the data in Drupal's fulltext index? (maybe not, as you can join the data at index time using *hook_nodeapi($op = "update index")* )

Maybe this means we have to think a bit more about the architecture, making it flexible enough to allow all the stuff we want to do =) And, right now, I feel a diagram or too wouldn't hurt in order to know what we're dealing with =)

tituomin’s picture

tituomin commented

Interesting problem, requires a bit of thinking! =)

When thinking about a solution for the enrichment functionality, it occured to me that you could even think about enrichment as a general feature of many kinds of Drupal nodes, not only Millennium Integration nodes. Of course, the enrichment will be dependent on what kind of metadata is available for a given node type, but even that could be handled with a good interface.

I agree that data from different sources should be kept separate to make data maintenance easier, and also about using the hooks to combine the data at load/index time.

In the short term, I can try make this functionality work with erasing any data. In the long term, we should have a better idea about overall architecture.

janusman’s picture

janusman commented
Status: Needs work » Fixed

Committed (a while back) a change so that what you mentioned in #11 (holdings info still being called while offline refreshing) will not happen anymore. We should continue discussion of aggregating metadata from other sources in #358765: Integrate with other external data sources (WorldCat, LibraryThing, Open Library)

tituomin’s picture

tituomin commented

Cool, thanks!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.