When I attempt to import 5000 records, it works out okay. When I try 10,000 records, I run into problems. I'm importing EndNote 8 XML; I'm pausing to validate the XML prior to import each time using Oxygen.

Sometimes, the screen just refreshes without giving me an error message of any kind and it doesn't import any records.

Other times, it imports but doesn't display a successful loading screen. Rather, the screen goes white when the process has concluded, and in checking the records I find that it imported only 8,676 records, even though the file was valid to the DTD.

Just to make sure it wasn't some odd character in the file, I made a 10,000-citation file by duplicating the content of the 5,000-citation file. It seems strictly to be a problem with volume.

My actual database contains 45K+ citations, so I'm trying to avoid breaking it down into small pieces and importing each piece.

Is there a known limitation on # of citations (or file size) for importing EndNote 8 XML? You can find my source files here:

http://staging.icpsr.umich.edu/drupal/temp/

As a side note, I can't seem to find an easy way (GUI) to empty the entire bibliography. When I import 10,000 records and have 1400 not show up, I would like to be able to easily empty the bibliography and start over.

-Matthew

Comments

rjerome’s picture

rjerome commented

Indeed you are sailing into uncharted waters at >10K. The most I have done is 2.5K but there is no fixed limit. The limiting factors will be the memory available to PHP and the length of time PHP is allowed to run any given script. I believe that out of the box, PHP can use 8M of memory and any given script can only run for 30 seconds but these values can be changed in php.ini.

Currently when importing files, I'm reading the entire file and parsing it into an array, then lopping through the array saving each individual entry as a new node. With 45K entries this array could get quite large.

If you want to wipe all the biblio entries, the easiest way is to disable the module on the module page, then go to the uninstall tab and uninstall it, then re-enable it on the modules page. That will remove all traces of biblio entries and reinstall clean tables (you might also want to try the -dev version for this since I made some significant speed improvements in the cleanup process recently).

I've now got copies of your files, so I'll do some testing to see if the process can't be fine tuned for large bulk imports.

Ron.

rjerome’s picture

rjerome commented

Well, I was able to import the 10K file (sort of...). I had to make a change to the code, increase the memory allowed to PHP and increase the max script time to 300 sec.

Then next problem is that the XML files you supplied have "ref-type" id's which don't match what I would expect. What version of EndNote generated that file?

Ron.

rjerome’s picture

rjerome commented

Ahhh.. It appears that the ref-types are mapped to EndNote 7 format types but the structure of the file is EndNote 8 (or greater) XML. These are the EndNote 7 mappings (which your files seem to use), the left most column is the EndNote id and the next one is the biblio id ...

   '0' => '102', // Journal Article
    '1' => '100', // Book
    '2' => '108', // Thesis
    '3' => '103', // Conference Proceedings
    '4' => '120', // Personal Communication
    '5' => '105', // NewsPaper Article
    '6' => '113', // Computer Program
    '7' => '101', // Book Section
    '8' => '106', // Magazine Article
    '9' => '100', // Edited Book
    '10' => '109', // Report
    '11' => '122', // Map
    '12' => '114', // Audiovisual Material
    '13' => '112', // Artwork
    '15' => '119', // Patent
    '16' => '107', // Electronic Source
    '17' => '117', // Bill
    '18' => '116', // Case
    '19' => '115', // Hearing
    '20' => '121', // Manuscript
    '21' => '110', // Film or Broadcast
    '22' => '118', // Statute
    '26' => '123', // Chart or Table
    '31' => '129' // Generic
rjerome’s picture

rjerome commented

Further to this, the EndNote8 mapping should look like this... (again the left value being the EndNote8 value and the right value is Biblio)

 $map = array(
  2 => 112, // artwork
  3 => 114, // Audio Visual
  4 => 117, // bill
  5 => 101, // Book Section
  6 => 100, // Book
  7 => 116, // case
  9 => 113, // software
  17 => 102, // Journal Article
  10 => 104, // Conference Proceeding
  12 => 107, // web page
  13 => 129, // Generic
  14 => 115, // hearing
  19 => 106, // magazine_article
  20 => 122, // map
  21 => 110, // film
  21 => 111, // broadcast
  23 => 105, // newspaper_article
  25 => 119, // patent
  26 => 120, // personal communication
  27 => 109, // Report
  28 => 129, // Edited Book
  31 => 118, // statute
  32 => 108, // Thesis
  34 => 124, // unpublished
  36 => 121, // manuscript
  37 => 129, // figure
  38 => 123, // chart
  39 => 129, // equation
  43 => 129, // electronic article
  44 => 129, // electronic book
  45 => 125, // online database
  46 => 126, // government_document
  47 => 103, // conference_paper
  48 => 129, // online multimedia
  49 => 127, // Classical Work
  50 => 128, // legal_ruling
  52 => 129, // Dictionary
  53 => 129, // Encyclopedia
  54 => 129, // Grant
matvey’s picture

matvey commented
Version: 6.x-1.0-beta9 » 6.x-1.x-dev

I'm still not having any luck getting it to work.

1) I had my network admin adjust php.ini.

/opt/apache/conf{8}# egrep "^max|^memory" php.ini
max_execution_time = 300 ; Maximum execution time of each script, in
seconds
max_input_time = 600 ; Maximum amount of time each script may spend
parsing request data
memory_limit = 512M ; Maximum amount of memory a script may consume

2) I adjusted my XML according to the new codes you sent. I'm now using the left column codes in post #4 above. I made three new files:

1-citation.xml
5K-citations.xml
10K-citations.xml

which can be found at the same URL in post #1.

3) I installed biblio-6.x-1.x-dev.

Unfortunately, I can't even import the 5K-citations.xml file. When I go to import the file I get a red box above the "Import file" form element that says "You must provide a file name" even though I've pointed it at the XML file. I get this error for the 10K and 5K file on Mac/Safari, Mac/Firefox, and Win/Opera. The 1-citation file works just fine.

Is there something I'm doing wrong?

rjerome’s picture

rjerome commented

I ran into that also, the problem is with the PHP "post_max_size", it has to be larger than the file size. I set mine to 10M.

Ron.

matvey’s picture

matvey commented

Thanks a ton for the fixes you made. I installed the latest patch and now it gives me the precise error message. I was able to adjust the file upload size in PHP and successfully upload the 10K citation files. So I now have a drupal bibliography of around 45K citations.

It's definitely sluggish, especially on the filter page, but I would say the 'import large files' issue has been resolved. Thanks again. :)

rjerome’s picture

rjerome commented

Yes, I thought those detailed error messages might be more useful :-)

Did you get the version with the batch upload that shows the progress bar as it goes?

rjerome’s picture

rjerome commented
Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.