An option to use b-identifiers as an alternative

I *think* you are incorrect in stating that the cart requires b-numbers; I have done some experiments using item numbers and it seems to work.

I *am* open to adding this feature, it would be great if we could mantain the option of using item numbers OR bib numbers, and not have it be a global option; perhaps some items were imported with i's, other with b's ...

Re: #3: After a large crawl there *should* be several items for some bib items; at least that's the result for our OPAC crawl =)

Re: #5; you are correct. The module was only storing the node-bibnum-itemnum relationship when nodes were created, but not updated.

Will commit this patch for millennium.module along with other fixes/tweaks.

@@ -1021,10 +1021,24 @@
   // Store node in MySQL
   node_save($node);
 
-  // Add relationship to millennium_node_opacitem table
-  $result = db_query("UPDATE {millennium_node_opacitem} SET bib_updated='now()', biblio_data='%s' WHERE bib_recnum='%s'",
-    serialize($node->millennium_biblio_data),
-    $bib_recnum);
+
+  // Add or update relationship to millennium_node_opacitem table
+  $result = db_fetch_object(db_query("SELECT * from {millennium_node_opacitem} WHERE item_recnum = '%s' AND bib_recnum= '%s' AND base_url = '%s'", $item_recnum, $bib_recnum, millennium_get_real_baseurl()));
+  if ($result->item_recnum) {
+    #drupal_set_message("millennium_node_update() updating millennium_node_opacitem");
+    $result = db_query("UPDATE {millennium_node_opacitem} SET bib_updated = NOW(), biblio_data='%s' WHERE bib_recnum='%s'",
+      serialize($node->millennium_biblio_data),
+      $bib_recnum);
+  } else {
+    // Add relationship to millennium_node_opacitem table
+    $result = db_query("INSERT INTO {millennium_node_opacitem} (nid, item_recnum, bib_recnum, item_updated, created, bib_updated, biblio_data, base_url) VALUES (%d, '%s', '%s', NOW(), NOW(), NOW(), '%s', '%s')",
+        $node->nid,
+        $item_recnum,
+        $bib_recnum,
+        serialize($node->millennium_biblio_data),
+        millennium_get_real_baseurl()
+    );
+  }

Just wanted to jot down a few notes...

Back to the issue... I would like to give users the option of being able to switch back and forth between one and the other mechanism.

So... thinking this:

• millennium_import_update_item() could optionally take in either the bib number, item number or both.
• When only the bib number is given, it only does logic to update/import based on that; the millennium_node_opacitem.item_recnum field would be left to null meaning the item number is unknown.
  • Note: If later we decide we do know the item number, and that item number does not exist, then null value is overwritten with the item number given to the function.
• When only the item number is given, it does the logic to determine the bib number, establishes additional relationships in millennium_node_opacitem (or overwrites a "null" value for item_recnum if one exists for that bib #)

The logic to determine when a node should be updated is not smart nor logical; I had an extra XRECORD call (that I recently removed) to get the bib record last update date, to determine if a node should be refreshed or not. This might have to be put back in (meaning a performance penalty of about 50%) or switch to an admin-selected expiry time (like, always refresh after 30 days).. or if crawling using items, decide that a new item for that bib means "refresh the node".

First part of the patch, nowhere near complete. LOTS of "TODOs" and dpm()s in the code (won't work if you don't have devel.module enabled)

Currently works:

Preview: http://example.com/millennium/preview/b123456
Manual import using lists of b-numbers: /admin/settings/millennium/manual-import
The tools, marc view, node view, etc. on imported records.

This patch changes the database schema... I think the Drupal cache needs to be cleared in order for it to pick up. I don't think I will come up with code to migrate from the earlier version to this (hey, this is DEV) =)

To test you have to start over importing some new records in /admin/settings/millennium/manual-import

This is a BIG patch, reworking Millennium to be friendly to b-numbers AS WELL AS i-numbers.

The gist is this:
* Crawling and batch importing accepts either item or bib record numbers.

How the node-bib-item relationship is handled:
* There is a 1:1 relationship between nodes and bib records (in the new table millennium_node_bib) for a single base_url (meaning importing same bib # for another base url will create another node)
* There is a 1:[0-n] relationship between nodes and items (meaning there can be 0 item records for a certain node). During import or crawl, specifying an item for import creates its bib and corresponding node first (if it doesn't exist), then the item gets linked via the table millennium_bib_item.

Handling of not-found items during crawl or batch import:
* trying a bib number that is not found in the WebOPAC will delete the node and the bib and item entries (if any).
* trying an item number that is not found in the WebOPAC will delete that item only; and if it was the only item left in millennium_bib_item it will also delete the node.

During crawl of batch import, items are marked "stale" after 30 days, and refresh whenever an import for that record is attempted after that time. If they are not stale, they are not refreshed (except if you specified so in a the batch import form).

Other changes:
* The batch import form is now in admin/content/millennium (we're adding content, so it seemed better than in admin/settings/X)
* Added a URL to directly create nodes using REST:
/millennium/import/[bib or item rec num]
e.g.
http://example.com/millennium/import/b123456

will create a new node based on that record. Or, if there is a node for that record already, it refreshed the bib. data in Drupal and redirects the user to that node. I want this as a bookmarklet (opening another issue for that) =)

TODO:
* The items data itself is still not of much use; this (as before) is just in place to allow future per-item data import; circulation stats, location data, etc. (presumably using XRECORD calls).
* Testing! Not tested at all: automatic delete when item or bib records not being found.

I failed to mention this requires uninstalling and re-installing the module, as new tables are needed. Again, no upgrade path is planned.

Set up the module normally, then you could quickly batch-import some items at
http://example.com/admin/content/millennium

Found a problem with the bookcart "finding" bib records even though they are not visible or do not exist when visiting the direct url (record=b123456).

The bookcart lists the item that doesn't exist with the title as the previous one in the list (that does exist), however some items are missing (like the year and the material icon)... so perhaps this can be asverted with some code.

Thanks for testing =) Keep the comments comming!

Ok, re: #16, we could unpublish the node when the last item for the node is not found (instead of deleting it). Finding a new item for that bib would then re-publish it.

However, then we could theoretically end up with lots of unpublished nodes, but I'm thinking we could just inform this somewhere on the reports page so the admin can take action if he wants (or, add an option that lets the admin delete those "orphaned" nodes). Thoughts?

re: #14: when giving a list of item numbers, I opted for the default format Millennium uses when exporting lists, which means it has a check digit (or X) at the end of the record number. However, I realize this is acually not consistent with the rest of the interface where we are asking just for the sequential number.

I guess I could add a checkbox above the list that says "ignore check digit (Use when exported record numbers Millennium contain check digits or 'X' at the end)."

This new patch addresses the issues found in #12, #14 and #16.

• Fixed the bookcart import; items not found were still being added to the bookcart under wrong bib numbers. The current fix might not be perfect; it only detects if a title is repeated and ignores the records for those repeated titles. This is more of an III bug, and probably needs testing to see if it's reproducible on other systems.
• Deleting nodes when items not found: it will still delete the node but only when this is the last item, AND if the corresponding bib record is also not found on the WebOPAC:
• Batch import via the list (textarea) has a checkbox to automatically strip off the check digits.

Te ease review this is a meta-patch comparing the patch from #19 to the one in #10. =)

#22: You're correct, just noticed this yesterday... guess I didn't actually test the listing functionality...

In fact, i've seen big record numbers (8 digits) like http://catalog.nypl.org/record=b18134823 ... it might be safe to allow from 5-10 digits.

I like the idea of having the regexps as constants...

Will issue a new patch soon... I think this is just about done, don't you think? =)

New patch. This addresses some issues from previous comments, and also some problems found during import using the bookcart.

* admin/settings/millennium has an option as to what action to take when bibs are no longer found.
* the cart wasn't correctly handling some WebOPAC configurations where the bookcart is paged. Fixed here.
* Added missing t()s inside drupal_set_message() (except some that are still there just for debugging).
* Added the regexps for recognizing records as constants.

Sorry for mixing up so many things in a single patch (this making it hard to review) but maybe at least it can be tested =)

Ah! There's a new function to test harversing: millennium_mass_fetch_test() that can be called from any PHP-enabled page that will scan some known OPACs and show results. It requires devel.module, though.

Committing this patch.

Tested with several OPACs, seems to be working fine =) Please report any problems you might see.

yay! =)