Bulk update biblio

You could enable Endnote export, and export your publications into Endnote. Then use Endnote to clean it up. They have a free trial version. http://endnote.com/

You can search in Endnote, but I don't think you can do search/replace. It's certainly faster to edit entries in Endnote that it would be using the Drupal interface, but for 20,000 entries, writing a script would probably save a lot of time.

You say you are going upload 20000 entries, what format is this? Just thinking you could do your edits before your first import?

I see, well as you correctly stated in your original post, working directly with the database could be a bit tricking if you stray outside the base "biblio" table, but depending on what you need to change, you might be able to work with the biblio_contributor_data and biblio_keyword_data tables independently, so long as you retain the cid and kid values in those tables.

Alternatively, if you really wanted to work in Excel, I could probably craft a CSV export/import feature which could include the nid and thus update existing nodes.

I guess one of the other questions is could the biblio nodes be modified on the drupal side AFTER the export but BEFORE the re-import? If so, then it gets a little more complicated because you would have to decide which one to keep. If not, the we could just replace the old node (which would essentially be an update). You also have the option of creating a "revision" which give you the ability to revert to the previous version if necessary.

With regards to Authors and Keywords, what sort of changes to you anticipate for these? The thing about authors is that only one instance of any given author is stored with pointers to all the publications the contain this author. So John Smith may be the author of 100 papers, but the author name is only stored once in the database. To complicate maters, there can be two or more similarly named authors if you know they are distinct, but again only one of each is stored. All this to say, it might actually be simpler to modify authors directly in the Biblio administrative UI since a single change there will affect all entries linked to that author. Basically the same story for keywords.

Just FYI, I'm already doing something fairly similar to this in the PubMed import module. It has the ability (via a cron job) to periodically query the PubMed server and look for updates to existing nodes and update them if if finds any. Similarly, I've written a bridge module for the XC OAI-PMH harvester with also does more or less the same thing, i.e. periodically updating existing nodes from an OAI server.

Hope that helps to clarify.

No I wasn't suggesting starting from one of the other modules, since they are vastly different for the most part. The actual update part is really quite simple, you just have to look at the incoming nid and then load the existing nid and merge the two and re-save it.

I see what you are getting at with the author names, and I know that this is the single biggest pain in the neck. I presume you are aware that the ADMIN UI has an author "merge" option as well which will allow you to pick one of those author variants and then globally change all the entries that use one of the other variants to reference the selected one. I typically would select the one with the most detail "Jerome, Ronald A.", then merge all the others with that. The UI will even highlight entries that have the potential to merge, so they are pretty easy to find.

If you know of specific errors in the XML, you might be better off to just do a search/replace in the XML file prior to the first import. It might save you a lot of grief in the long run.

Are you sure that you need to keep the NID numbers consistent across time while you are updating and editing? Just asking.

We periodically wipe our entire database and replace it with a freshly exported version from our offline files. In our latest version, we are using a custom field that saves your own unique ID number (you could possibly use an auto-generated record number in Excel to start with). Then we use that number to create a citekey (something like bib#######, where the ######## is the record number ID). And we generate our biblio URLs with pathauto, using that citekey value as part of the URL (so, for e.g., you could end up with a URL like example.org/biblio/bib####### for each). Then rather than updating each record and trying to maintain the NID numbers, you would just re-import the whole database of 20,000 each time. The NID numbers would change but your URLs and citekeys would remain consistent as long as you kept that custom field intact across each version.

Maybe you need the NID numbers because you are cross-referencing them elsewhere?

Phil.

Another quick thought on this, which may or may not help the original poster.

There is a Biblio Advanced Import module for biblio 6.x that is supposed to provide the ability to merge or update pre-existing biblio records, based on a defined set of duplicate field matching criteria. I haven't tested this module, and it looks like it stalled in beta stage, so it may not be as robust a solution as you might like, but it may meet your needs if you are just going through this major editing process once, over the next year.

We may be testing this module for compatibility with biblio 6.x-2.x in the coming months.

Phil.

Just a quick follow-up on this for the purpose of completeness and archiving of support answers.

We are using Biblio Advanced Import with biblio 6.x-2.0-rc2 to import the same records three times during our workflow to produce a final record. Each successive import is set to use a custom field as a citekey to determine duplicates, and the custom field holds a copy of the unique Reference Manager ID value. Biblio Advanced Import has been configured to replace any fields that previously were empty, but to leave other data in place, and to create a new revision of the node. This all seems to work as designed for us (!).

If the OP happens to check back into this thread, you should be able to do the same thing as us, but configure Biblio Advanced Import to replace the record entirely when a duplicate is found: this would update all fields with whatever your new values were, while retaining the biblio node ID (but changing the revision ID).

Phil.

This version is no longer maintained. If this issue is still relevant to the Drupal 7 version, please re-open and provide details.