These seem like separate issues. I need to refactor the code but first it must be decided whether or not to use hook_feeds_after_import. This could allow batching to be used +1.

Comments

twistor’s picture

twistor commented
Priority: Normal » Critical

After some deeper investigation the odds of a timeout are huge if someone is trying to crawl greater than ~40 pages. Batch support is a must.

twistor’s picture

twistor commented
Title: Refactor code/Decide whether or not to use hook_feeds_after_import » Add batch support.
twistor’s picture

twistor commented

I was wrong on all accounts. Implemented initial batching support. It doesn't however support Feeds batching, it just makes each page a job. Batching batches makes my brain explode.

twistor’s picture

twistor commented
Category: task » feature
twistor’s picture

twistor commented
Version: 6.x-1.x-dev » 6.x-1.0-beta1
twistor’s picture

twistor commented
Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

sanguis’s picture

sanguis commented
Version: 6.x-1.0-beta1 » 7.x-1.0-beta1

I could also use this for 7. ine page per cron is too small

chrisroditis’s picture

chrisroditis commented

Hello, any updates on this? Is there a temporary workaround that could allow importing more than one page per cron?

twistor’s picture

twistor commented
Assigned: twistor » Unassigned