Hi,

I have a problem with either timeouts or fatal memory errors.

i was wondering if I could use job queue (http://drupal.org/projects/job_queue) to meter out when the feeds are done so that I can get through all the feeds.

Thanks,
Chris

Comments

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

Can you explain me how job_queue can help to solve memory and timeout limit issues?
You did not mention what's the actual value of the php limits (max_execution_time and max_allowed_memory. Maybe these values are simply too low.

socialnicheguru’s picture

socialnicheguru commented

I am not sure how cron runs with feeds.
Does it run until it times out and then the next time it runs it just starts all over again instead of going in sequence?

for testing I made them ridiculously high 650M and 200s and I still got errors.

For feeds that might take a long time or memory, I woudl like to run them in chunks.
so if my feed has 1000 elements, i would like cron to run on the feed so that the first run would give me 1 to 100 (or whatever i could process within the cron limit), the next run would give me 101 to 200, etc.

Jobqueue would enable me to run conr as I described above.

Plazmus’s picture

Plazmus commented

I don't know how cron runs with feeds, but definitely it's trying to do as much as it can, so job queue looks like a perfect solution for websites which have a big number of feeds to scan.

I think the best scenario would be (if possible):

- run feedapi script from cron
- feedapi script will check which feeds need to be updated
- put all feedapi items that should be created into the queue, there should be a loop that will go through all items one by one and if will finish then wait for another cron run (probably job queue can handle this)
- next cron run checks whether feedapi script is running if yes then just skip if no run script

This is just my imagination how I would like to see feedapi process of updating feeds, I started playing with feeds aggregation on drupal this week, so any comments/thought from more experienced users are very welcome :)

alex_b’s picture

alex_b commented
Status: Active » Closed (won't fix)

FeedAPI updates as many feeds as possible in the time allocated for it (see Cron time for FeedAPI [%] setting on amin/settings/feedapi).
If you're running into memory limits, increase the PHP memory limit (500 MB and more can be necessary especially when you're using feedapi_node processor).

FeedAPI even stops processing in the middle of a feed when it's time is over.

I am not intimately familiar with Job Queue but I really doubt that it can do much for FeedAPI here: Job Queue must handle scheduling in some similar way, stopping processing in the middle of a feed is going to become next to impossible and FeedAPI's egrigious memory consumption with processors like feedapi_node is something that needs to be fixed in FeedAPI (ideas welcome here, I couldn't find a way to reliably release memory in PHP yet).

That said, there may be a bug in FeedAPI that causes these timeouts you're experiencing. I'm very interested in helping you solving them. I'd encourage you to dig deeper into FeedAPI and investigate the issue: which feeds cause the timeout? do they cause the timeout by themselves? What number of feeds does start to cause the timeout? Does the site have enough memory? Does it matter which parsers are enabled? Is it an issue with feedapi_node (use as alternative http://svn3.cvsdude.com/vvc/devseed/sandbox/drupal-6/feedapi_fast/ ).

Please report issues to #370318: Keep exceeding cron time with FeedAPI. I am closing this issue, as the original question (Job Queue) is answered.

socialnicheguru’s picture

socialnicheguru commented

does feedapi start updating from where it left off or does it start from the beginning again?

I have a site with lots of feeds. I would prefer that it updates from where it left off. I would not get all data upated at once, but I can be assured that my feeds will eventually update.

Thanks,
C