Testing Feeds on two servers, I noticed that the number of jobs sent to Queue seems to grow without limit.

With 94 feed sources set to import a small Json file that creates one node every 1 hour, the number of jobs sent to Queue has grown from 83 jobs to 972 jobs in sixteen hours. Same thing happens on a server where I have close to 1500 feeds importing RSS, JSON and CSV feeds, the number of jobs sent to Queue has grown from 201 jobs to 6156 jobs in sixteen hours.

When I look at Feeds log, I see that only 4 to 10 imports get performed when I run cron which could explained why jobs are pilling up in Queue.

I've noticed that sometimes the number of jobs in Queue does go down a bit which tells me that this is a different issue than #762876: drupal queue integration not clearing itself?

My config is:
-feeds_cron, job_scheduler_cron, and queue_ui_cron set to run every 15 minutes with elysia cron
-Drupal 7.7

Comments

florentcm’s picture

florentcm commented

Subscribe.

sylvanos’s picture

sylvanos commented
Category: bug » feature

I solved the problem and unmarked this as a bug (sorry) but rather a feature request as the solution I found involves changing stuff in feeds.module.

The reason why jobs keep pilling up in Queue is that every time cron runs not enough jobs are processed. This is due to a time limit of 15 seconds I found in this function:

function feeds_cron_queue_info() {
$queues = array();
$queues['feeds_source_import'] = array(
'worker callback' => 'feeds_source_import',
'time' => 15,
);

I've increased the time limit to 45 and voilà: 3 times more jobs are processed and the Queue is kept reasonably low.

I'd like to ear from the developers on this one because
1. If changing this time limit is a bad idea, I wouldn't want anyone else to do it.
2. I'd like to have their opinion on making this change more obvious so people find it faster than I did

js’s picture

js commented

Hmmm, I just discovered queue and over a million records. I obviously have some learning to do.

I will try increasing the time and monitoring the queue, but any suggestions would be great.

js’s picture

js commented

Changing the time to 45made all the difference for me. I will experiment over time.

I am running cron with drush frequently.

Slim Pickens’s picture

Slim Pickens commented

Thanks for the tip.

My site aggregates about 90 feeds with feed update at 10 minutes and cron set at 5 minutes. There have been a couple of issues arising lately of some feeds not updating so I was checking out the database. I found the queue table had 41K jobs.

I changed the time to 45 as described above, reduced the feed update time to 30 min and upped cron to 2 minutes. Over the last day or so the queue table has been reduced to about 5K and should be up to date in a day or so.

Clearly these timing values need to be fine-tuned for each site/host combination.

It would be good if the Feeds Admin UI provided the means to change the time value as above.

js’s picture

js commented

I would be happy with hooks so that I can override the defaults and they survive updates.

janip’s picture

janip commented

js: You could alter the time by implementing this hook?:

http://api.drupal.org/api/drupal/modules!system!system.api.php/function/...

js’s picture

js commented

Of course! Thank you @janip, I appreciate the lesson tip. Jerry

megachriz’s picture

megachriz
Primary language Dutch
commented
Category: Feature request » Support request
Issue summary: View changes
Status: Active » Fixed

#7 answers the question, here is an example for increasing the time limit for the feeds_source_import queue:

/**
 * Implements hook_cron_queue_alter().
 *
 * Changes runtime limit for feeds_source_import queue to 90 seconds.
 */
function mymodule_cron_queue_info_alter(&$queues) {
  if (isset($queues['feeds_source_import'])) {
    $queues['feeds_source_import']['time'] = 90;
  }
}

By the way, this queue now has a time limit of 60 seconds in the module.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.