Millennium OPAC Integration version 6.x-2.0.

I can't batch import records from our OPAC (library.cca.edu). The message says "Crawl http://library.cca.edu: 5 crawled, 0 pending (0 found, 5 not found) 57.3 items per second"

I have tried every method (list of records numbers, range of record numbers, featured list URL and key word search), but I get the same results with all.

However, I can successfully batch import records from other Libraries' OPACs (e.g., http://www.consuls.org).

I discovered that I can import individual records from our OPAC using the preview technique (http://libraries.cca.edu/millennium/preview?url=http://library.cca.edu/r... and then clicking on the link to import that record)

I can also successfully batch update existing records using the "Do not create items, only update existing using local MARC data" option.

We definitely have MARC format enabled for exports.

I first encountered this problem on our live site which is still on version 6.x-2.0-alpha, so I just upgraded our development environment (which also has this problem) to 6.x-2.0, but the behavior remains the same.

Could the problem be caused by the fact that the base URL for our OPAC (library.cca.edu) automatically redirects to our Library website (libraries.cca.edu)? It seems to me this might be what makes the keyword search method fail (is there any way I can specify our true OPAC base URL of "library.cca.edu/search" without the Millennium module stripping everything after .edu?). But I would imagine the redirect has no effect on the URL method as I'm linking deep enough into the OPAC to avoid the top level redirect - no? I link to featured lists like this: http://library.cca.edu/search~S0?/ftlist^bib38%2C1%2C0%2C23

I can see from my successful batch imports from other library OPACs and from the few successful individual imports from our OPAC, that the fields and taxonomy terms I set up for my biblio content type are receiving the data as mapped, so I don't *think* there is anything wrong with my content type and mappings.

Any ideas or suggestions out there?

Anne

CommentFileSizeAuthor
#10 bib_display.pdf47.11 KBbanoodle

Comments

janusman’s picture

janusman commented
Assigned: Unassigned » janusman

I would need to take a look, there could be some extra or different HTML markup that is causing things to not be parsed on the MARC exports.

This has been tested pretty heavily with a few different OPACs, but there could always be an edge case that might need to be included on the parsing code.

There's a hidden option in code in millennium.import.inc that enables a pretty extensive set of debugging messages, here:

function millennium_fetch_records_via_bookcart($recnums, $complete_holdings = FALSE, $base_url) {
  // Set the following to TRUE for lots and lots of debugging info =)
  $debug = FALSE;   // HERE!

Setting that to TRUE might help you track things down.

I will also take a look and see what I can come up with.

banoodle’s picture

banoodle commented

Thanks! I enabled debugging and then attempted importing 20 records from this featured list:

http://library.cca.edu/search~S0?/ftlist^bib37%2C1%2C0%2C82/mode=2

In the interface, I got a HTTP error 500.

Under recent log entries, I'm seeing these errors...

5 of these errors:

Table 'acquiadrupal6.millennium_node_bib' doesn't exist query: SELECT bib_recnum FROM millennium_node_bib WHERE nid=188 in /etc/acquia-drupal/6/sites/all/modules/millennium/millennium.module on line 2419.

Followed by 3 pairs of these errors...

Table 'acquiadrupal6.millennium_node_bib' doesn't exist query: SELECT count(*) FROM millennium_node_bib in /etc/acquia-drupal/6/sites/all/modules/millennium/millennium.admin.inc on line 528.

Table 'acquiadrupal6.millennium_import_queue' doesn't exist query: SELECT count(*) FROM millennium_import_queue in /etc/acquia-drupal/6/sites/all/modules/millennium/millennium.admin.inc on line 529.

banoodle’s picture

banoodle commented

Also attempted import with keyword search=edo.

This also failed with a HTTP error 500 and similar errors in the log file.

However, under Recent Log Entries, there was this additional message at the top of the screen...

Added items to bookcart response: path = http://library.cca.edu/search*eng/X//1,1,1/?save=b1039123&save=b1032004&...

banoodle’s picture

banoodle commented

Hmm - I can see that none of the database tables in millennium.install got created except for millennium_import_cache (which is empty).

I tried running update.php again and selected 6001 from the pick list for Millennium Module. I get these errors...

user warning: Table 'acquiadrupal6.millennium_import_queue' doesn't exist query: ALTER TABLE millennium_import_queue ADD `base_url` CHAR(80) DEFAULT NULL in /usr/share/acquia-drupal6/includes/database.mysql-common.inc on line 307.
user warning: Table 'acquiadrupal6.millennium_import_queue' doesn't exist query: ALTER TABLE millennium_import_queue CHANGE `base_url` `base_url` CHAR(80) NOT NULL in /usr/share/acquia-drupal6/includes/database.mysql-common.inc on line 529.
user warning: Table 'acquiadrupal6.millennium_node_bib' doesn't exist query: ALTER TABLE millennium_node_bib CHANGE `biblio_data` `biblio_data` MEDIUMTEXT NOT NULL in /usr/share/acquia-drupal6/includes/database.mysql-common.inc on line 529.
user warning: Table 'acquiadrupal6.millennium_node_bib' doesn't exist query: SELECT DISTINCT(base_url) FROM millennium_node_bib in /etc/acquia-drupal/6/sites/all/modules/millennium/millennium.install on line 367.

banoodle’s picture

banoodle commented

OK - I followed the advice here for re-installing modules: http://drupal.org/node/66749, and now the proper millennium schema exists in my database.

I tried importing the featured list again. I still get a HTTP error 500, but this time it is also saying "An error occurred while processing millennium_batch_import_crawl_operation with arguments millennium_batch_import_crawl_operation"3

banoodle’s picture

banoodle commented

When I first reported this problem, I could batch import from other OPACs and I could use a link like this to import individual records from our OPAC: http://vm-lib-www-dev-01.cca.edu/acquia-drupal6/millennium/preview?url=h...

But now I can't get any importing to work. On the preview screen, the "Import this record now" link is crossed out and no longer works.

Attempting to batch import from another OPAC is now failing with the same message...
"An error occurred while processing millennium_batch_import_crawl_operation with arguments millennium_batch_import_crawl_operation"

No other error or status messages are being written to "Recent log entries"

banoodle’s picture

banoodle commented

One other observance: I just attempted clearing drupal cache and I got this message which I have never seen before: The following enabled modules are incompatible with aggressive mode caching and will not function properly: autoload, dbtng, ldapauth, millennium.

banoodle’s picture

banoodle commented

Turning off debugging got me back to where I could import other OPAC records.

I finally got batch import to work by commenting out the javascript we have in bib_display.html.

What you said about checking how the code parses the html made me think of this.

So, I suppose we could temporarily comment out our javascript each time we import more millennium records, but is there an alternative solution?

I'm attaching our bib_display.html. The offending javascript is commented out (lines 98 thru 102).

Thanks so much for your help!

Anne

janusman’s picture

janusman commented

I think you missed uploading the attachment :-)

banoodle’s picture

banoodle commented
StatusFileSize
new bib_display.pdf47.11 KB

Doh!- here you go. - Anne

janusman’s picture

janusman commented

On your opac you don't have the option to output a saved list in MARC format to the screen. It looks like this is the missing piece of the puzzle.

This illustrates the missing option:
http://i.imm.io/Y7QU.png

Do you think you can turn it on and then try testing again?

banoodle’s picture

banoodle commented

You're right! I had enabled the browser/screen option on Development when I was grasping for straws, but later forgot I had done that.

So, the javascript wasn't the problem after all. I guess I commented it out around the same time I enabled the browser/screen option on Development.

I'm embarrassed because I read all the bloggers saying it's important to have export as MARC record enabled. I didn't realize you need to enable it and also enable the export to browser/screen option.

Thanks for clarifying!

LOVE the module!!!! It's so cool!

Anne

banoodle’s picture

banoodle commented
Status: Active » Closed (fixed)

Closing this. Wish there was a status for Noob Idiocy ; )

janusman’s picture

janusman commented

Thanks for the kind words! =)

SO, I guess this means we can close the issue. If you have some other issues please open a new one (unless it's about the import breaking).

janusman’s picture

janusman commented
Status: Closed (fixed) » Closed (works as designed)

OOps setting to "works as designed".

As for me, it probably wouldn't hurt to add a check to see if the 'Screen' option is there and throw a warning if it's not there.

janusman’s picture

janusman commented
Issue summary: View changes

removing port number (2082) from crawl message example (our test OPAC is at library.cca.edu:2082). I have this problem on our live and test sites, so I'm removing the port number to avoid confusion.

webster h.’s picture

webster h. commented

Hello janusman,

I have this problem, too.

I use the millennium module to import the bib records from our library.

It works fine until yesterday. But we have a system upgrade yesterday.(millennium to sierra)

And now, it shows ->warning: Division by zero in /var/www/millennium/sites/all/modules/millennium/millennium.admin.inc on line 1018.

I could not find any way to solve it.

I try to import from other library's OPAC. It's working.

So I check the export part. It dose have a "screen" option.

And the OPAC looks the same as it was upgraded. I don't know if there's anything I missed or ???

Hope you could give me some help.

This module is really important to me.

Look forward to your reply.

BTW, our library's OPAC url is http://140.136.208.1/*eng
(In the millennium settings, I only set http://140.136.208.1)

janusman’s picture

janusman commented

Hello @ALOHA!

Do you suppose you can follow my advice from the first comment on this thread?

Namely, to briefly hack this function:

function millennium_fetch_records_via_bookcart($recnums, $complete_holdings = FALSE, $base_url) {
  // Set the following to TRUE for lots and lots of debugging info =)
  $debug = FALSE; // NOTE: Requires enabled devel.module

by setting debug to TRUE, then running the import again, and attaching the errors you see as a file or copied into your comment?

Thanks!

webster h.’s picture

webster h. commented

Hello,

Thank you for your reply.

I found the reason...
I was blocked by the system due to query too much data.

BTW, it's really a wonderful module!

janusman’s picture

janusman commented

Good to know! Thanks for the comment :)