I'm running XC 6.x-1.1-alpha5, and I can no longer ingest via OAI. Jobs that worked fine before now fail, Apache fails with a 500 (?)

172.16..... - - [06/May/2010:18:40:18 +0000] "GET /harvester/schedule HTTP/1.1" 500 458 "http://sandbox.citebank.org/harvester" "Mozilla/5.0 (X11; U; Linux
i686; en-US; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3"

Then Drupal errors say that 'Cron' run exceeded the time limit (but nothing has been changed in the php.ini) - AND - earlier versions of XC running on the same host still work when I run the same jobs...so I'm not sure where to try and figure out what's wrong.

warning cron 05/06/2010 - 13:27 Cron run exceeded the time limit and was aborted. admin
error php 05/06/2010 - 13:27 Invalid argument supplied for foreach() in ... admin
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.182 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.182 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.182 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.181 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.181 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.181 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.180 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.180 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.179 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.179 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.179 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.178 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.178 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.178 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.177 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.177 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.177 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.176 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.176 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.175 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.175 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.174 oaiharvester.batch.php:109 (oaiharvester_cron())

farther (pages) back I'm seeing more, notice these are all at the same time...

cron 05/06/2010 - 13:27 launch ... admin 13:27:15.026 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.026 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.025 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.025 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.024 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.024 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.023 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.023 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.022 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.022 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.021 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.021 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.020 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch http://biodiversitylibrary.org/oai,mods,title, admin 13:27:15.020 oaiharvester.batch.php:109 (oaiharvester_cron())
cron 05/06/2010 - 13:27 launch ... admin 13:27:15.020 oaiharvester.batch.php:109 (oaiharvester_cron())

Comments

phil.cryer’s picture

phil.cryer commented
Status: Active » Closed (fixed)

ok, have "fixed" this by installing it on a fresh install of Drupal - unsure how we're going to get it working no our current install, but will start by unistalling the modules and dropping all associated database tables.

phil.cryer’s picture

phil.cryer commented
Status: Closed (fixed) » Active

...now that I just closed it I tested it running purely from the cron hook, and we get the same error! Again, we can run it manually and force a harvest, which will work, but when calling it via cron it fails. After a bunch of 'launched' messages in the logs (like what I have above) it fails with an invalid argument error. Full message is:

Invalid arguement supplied for foreach() in /var/www/test3.citebank.org/sites/all/modules/xc/xc_util/xc_util.module on line 476.

again, this is on a fresh install of the latest Drupal (6.16) and XC alpha5

pkiraly’s picture

pkiraly commented

Hi Phil,

Sorry for the late answer. Could you send me the URL of the OAI provider you harvest?
In generally I am working on the dev version, you can get from the CVS. Soon we will release another alpha version, but I'd like to try whether this problem exists inside the dev version.

Regards,
Péter

pkiraly’s picture

pkiraly commented

Hi Phil,

I found a problem when harvesting ZooKeys OAI-PMH provider (http://pensoftonline.net/zookeys/index.php/journal/oai?verb=ListRecords&...). If the dc:date is set to be indexed in Solr as date type field (which is the default setting in Drupal Toolkit), Solr raised an exception and prevented to add the records into index, because the date in not in Solr compatible date format. Now in the dev version I fixed it (http://drupal.org/cvs?commit=365742). Please let me know if you harvest this provider, or another one.

Péter

phil.cryer’s picture

phil.cryer commented

Thanks for looking at this, I used the CVS version, but still had issues. I installed a stock Drupal 6.16 (so new install, fresh database), then used the current xc-HEAD. Anytime I have the XC submodule OAI Harvester (with or without the XC OAI Harvester submodule) enabled (NOTE: just enabled, no OAI repos even added yet), a cron called from crontab fails with the error message:
"Cron run exceeded the time limit and was aborted."

From Apache's access log:
111.222.333.444 - - [12/May/2010:19:40:11 +0000] "GET /cron.php HTTP/1.0" 302 615 "-" "Wget/1.11.4"
111.222.333.444 - - [12/May/2010:19:40:12 +0000] "GET /?q=batch&op=start&id=7 HTTP/1.0" 200 3381 "-" "Wget/1.11.4"

Note that if you have XC OAI Harvester enabled it automatically enables OAI Harvester, but it is possible to just have OAI Harvester enabled, either of these two ways result in the same errors. So something in OAI Harvester is causing cron to timeout and fail. Also, when I was testing on an old install I saw that cron was still telling XC to pull from OAI repos that I had deleted, and that's what was causing all of the "launch ..." errors from above, so that's another issue.

Also, just in case you want to test against them, these are the initial 3 OAI repos that I'll be pulling data from:
http://pensoftonline.net/zookeys/index.php/journal/oai
http://www.scielo.br/oai/scielo-oai.php
http://beta.biodiversitylibrary.org/oai

I'm happy to test anything to help, and I'll poke around the OAI Harvester submodule as well. Thanks.

flaviovs’s picture

flaviovs commented
Version: 6.x-1.1-alpha5 » 6.x-1.1-alpha8

Apparently this error comes from a buggy function of the xc_util module. The fix:

 function xc_util_array_unique($input) {
   $serialized = array();
-  foreach ($arr as $item) {
+  foreach ($input as $item) {
      $serialized[] = serialize($item);
   }
   $serialized = array_unique($serialized);
 
   $output = array();
-  foreach ($output as $item) {
+  foreach ($serialized as $item) {
     $output[] = unserialize($item);
   }
   return $output;
}

Sorry for not providing a real patch, but the change is straighforward.

Apparently this also got rid of some duplicated nodes that were being imported (I'm using only the OAI Harverster to feed Dublic Core to CCK).

gmontanez’s picture

gmontanez commented