Jump to:
| Project: | Millennium OPAC Integration |
| Version: | 7.x-2.0-alpha1 |
| Component: | Importer/crawler |
| Category: | support request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Issue Summary
I set this up this last fall and it worked great -- I have just gone to do updates, however, and I find that the system is not importing nodes, and I'm not sure what's going on. I have tried entering the featured list url (which is how I created nodes originally), I tried a record number listing, and reimporting already existing records. Each type crawls the correct number of records (so I know it is going to the right place), but then it shows that number not found, and it does import any nodes. I double checked permissions and settings and everything looks good, I'm not sure where the problem is.
This is the message I get:
* Crawl http://lis.lis.wwu.edu: 1 crawled, 0 pending (0 found, 1 not found) 2.1 items per second
* Node import http://lis.lis.wwu.edu: 0 processed, 0 pending (0 created, 0 updated, 0 untouched, 0 errors) 0.0 items per second
* Total time elapsed: 0.5 seconds
Any suggestions on where else to look for problems would be appreciated!
Comments
#1
Today I tried setting it up to do an automatic crawl, and I receive this message:
* warning: array_fill() [function.array-fill]: Number of elements must be positive in /srv/data/apache2/hosts/library/htdocs/includes/database.inc on line 253.
* warning: implode() [function.implode]: Invalid arguments passed in /srv/data/apache2/hosts/library/htdocs/includes/database.inc on line 253.
* user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') GROUP BY n.nid ORDER BY n.nid LIMIT 0, 50' at line 1 query: SELECT n.nid, n.title, SUM(IF(td.vid = 5, 1 ,0)) AS num_terms FROM node n LEFT JOIN term_node tn on tn.nid = n.nid LEFT JOIN term_data td ON tn.tid = td.tid WHERE n.nid>0 AND n.title>'' AND n.type in () GROUP BY n.nid ORDER BY n.nid LIMIT 0, 50 in /srv/data/apache2/hosts/library/htdocs/sites/all/modules/hilcc/hilcc.module on line 946.
It's very odd -- everything worked fine a few months again when I first installed these modules. I keep uninstalling modules I have added since then, but I am not able to fix the problem...
#2
Ok, comment #1 seems to be related to HILCC module, could you disable it for now and see what happens? (Or I *think* you could fix it by just assigning node types to the HILCC vocabulary... but maybe not) =)
#3
I tried assigning a node type to the vocabulary -- still didn't work.
I tried disabling hilcc module -- still didn't work.
I tried disabling and uninstalling hilcc module -- still doesn't work.
Other ideas?
#4
The error message are gone, by the way, but I'm still not getting records imported.
#5
--I just rechecked my settings -- they look fine.
--I tested importing an individual record using the preview function and it worked fine.
--I verified that there are no other new modules installed since I last used this -- I already disabled and uninstalled the one new addition.
--I set all my (content type) book permissions to allow anonymous and authenticated users create, edit and delete permissions (just in case).
--Nothing appears in the error log when attempting to do a batch import
Individual record import also worked fine in my production site where I haven't made any changes to hilcc or other modules, but no batch import in either system.
I'm kind of at a loss for what else to look at / test / change / check...
#6
It would be great if I could get your configuration...maybe a dump of your variable table. If that can't be, youcould tell me your source OPAC and record numbers you are trying to import.
Also tell me what happens when you try to import a single record using the single-import URL (e.g. http://yoursite.com/millennium/preview?url=http://youropac.com/record=b1... ... of course editing the "yoursite.com", "youropac.com" and record number)
#7
BTW, Another way to diagnose is to try to import items using the demo site here: http://drupalmty.com/millennium/
#8
OK -- same thing happens on the demo site. I tried both a featured list import and a record number range -- it crawls the list, then reports that they were not found:
* Crawl http://lis.wwu.edu: 786 crawled, 0 pending (0 found, 786 not found) 15.1 items per second
* Node import http://lis.wwu.edu: 0 processed, 0 pending (0 created, 0 updated, 0 untouched, 0 errors) 0.0 items per second
* Total time elapsed: 0.0 seconds
I AM able to import single records using the single-import URL.
Would it still be useful to have configuration information, or can we safely assume something is not right perhaps at the III level, since I can't import into the demo site either? It is odd that I can import a single record, but not a batch of records...
#9
I would like to chime in with this exact same situation is happening to me. I had successfully installed this module in a previous test version of my site months ago. A couple weeks ago I installed the latest version and now have the exact same situation - can't get it to batch import but it will import single records. Exact same message as reported above (crawls lists, but says 0 found, 0 imported). Both with my catalog & with demo site.
#10
I think I found the problem.
For batch imports, the module requires the "MARC" output option be available on the "Email/Export Saved Records".
@magbird: I confirmed this is not enabled on lis.wwu.edu; could you try enabling it and see if that fixes it?
@lwiegand: could you confirm that the MARC export option is currently disabled, and perhaps try enabling it and see if that now works?
On my part, I think this should be checked for when you're adding source opacs to the module (or on manual batch imports). There could be an option to fall back to try to import using individual MARC views on records, but that would be waaay too slow.
#11
Thanks! Yes, that fixed it. We did a system update since that time, so something must have changed here. Thanks for your help. I am back in business!
#12
Alright! Leaving open to document and also code a basic check.
Changing issue title.
#13
D7.14
PHP 5.3
Millennium 7.x-2.0-alpha1
The change to the Marc output option allows my site to import single items from our catalog - catalog.lancasterlibraries.org. Prior to this change, I could not import single items.
However, my site now has the same symptoms of the previous users. Perhaps it's related to the Alpha release, or not, but the similarities are a little to close, so I had to comment and see if anyone else was having the same issues or found any other solutions.
We cannot import batch number ranges. The keyword query works. The URL for a result listing works. The Record number listing works.
#14
Maybe this is a different issue, please do not rename issue titles, since it reflects what the actual problem was. =) In your case, did you try using the demo site mentioned in comment #7? Try and see if you can import items from your OPAC thru that site. Please report back if you can or can't, and copy here any messages you see =)
Another tip would be to try out the 7.x-2.x-dev version, which has a few additional bugfixes.
Good luck!