Hi,

I try to import 5000 Dublin Core registers but its only import by hundreds.

I created a new feed importer
- I attached the importer to the Feed node
- I set the fetcher to HTTP OAI-PMH Fetcher
- I set the parser to Parse Dublin Core (oai_dc) metadata from OAI-PMH repositories
- I set the processor to Node processor

Also try import Taxonomy but not worked.

Cheers

CommentFileSizeAuthor
set_options.png116.07 KBpuhig

Comments

janusman’s picture

Your initial settings you mentioned above are correct. The 100-item limit is, for now, a hardcoded limit on the module repository response.

To import more items, you can do this:
* Go to admin/build/feeds and edit your importer configuration
* Under "Basic settings", on "Minimum refresh period" choose "As often as possible"

Now, you can import all items one of two ways:
1) Run cron until you import all records.
or
2) Go to your created Feed node, and click on the "import" tab. (You should see a "Import all content from feed?" confirmation page... which you should confirm :) )

Leaving this open as this needs fixing and/or clarification. Perhaps there should be a limit that could be set upon importer creation (alongside where you set the Repository and Set).

puhig’s picture

What parameter controls this ? I want import all in one time

janusman’s picture

In reality, it's not a single parameter to change; it's more about the architecture of the whole Feeds OAI-PMH module.

The "fetcher" part of the module tries to be only that... a fetcher. This means it doesn't try to parse anything from the response (except the resumptionToken, which it just stores for the next run). This means that the 100 records you are seeing is not because of the module specifying that number, but because that is the number the repository that you are harvesting decides to return as a maximum on a listRecords request.

Now, if the fetcher indeed tried (as you suggest) to bring in, say, 30,000, 300,000 or more records and THEN pass that on to the next steps in feeds (parser, processor, etc) then we'd surely have problems:
* all those records have to fit in memory. Thus, PHP could throw errors when that limit is reached.
* parsing is resource intensive. At least the way I'm doing it (using SimpleXML), it requires a good deal of memory and those functions are known not to be very scalable.

So if there was a way to get around the above, THEN we can figure out a way to bring in *everything* in a repository/set. You can see this isn't simple (contributions welcome!)

So, again, the workaround is to follow the steps from #1.

Leaving open for any ideas and code. I'm not very inclined to do it as it doesn't seem high priority (again, because of the available workaround in comment #1)

janusman’s picture

For now, some workarounds:

You can basically do it by hitting the "import" tab until you get all the items, or wait out for Cron to run.

An alternative could also be for you to use any of the different cron modules out there that can let you invoke certain modules' hook_cron() more frequently than core; for example SuperCron or Ellisya Cron.