I have made some changes to the Drush integration to allow the queue to be run a little differently other than just calling the drupal_queue_cron_run() just once.
I have made it so you can run it 3 different ways.
1. As it is now, run once and exit, or n times.
2. Run forever.
3. Run until the all the queues have no jobs.
ATM I am using option 3 to process a ton of jobs that need to be caught up on.
| Comment | File | Size | Author |
|---|---|---|---|
| #10 | 1020946-10-drupal_queue-repeat-sleep.patch | 2.67 KB | neilnz |
| #1 | 1020946.patch | 2.56 KB | gordon |
| drush.changes.patch | 2.72 KB | gordon |
Comments
Comment #1
gordon commentedI have fixed up the patch so that it will pass the tests and apply easier.
Comment #3
Anonymous (not verified) commentedsubscribe.
Comment #4
sdboyer commentedsubscribe, and tagging as this may be crucial for phase 2
Comment #5
sdboyer commentedUntagging. We got around it.
Comment #6
gordon commenteddrush.changes.patch queued for re-testing.
Comment #7
gordon commented#1: 1020946.patch queued for re-testing.
Comment #8
neilnz commentedThis is similar to what the DataSync (http://drupal.org/project/datasync) module does - it starts a daemon on cron that forks into multiple workers and they process the queue at a limited concurrency level until it's empty, then the forks exit back to the main daemon and it polls for jobs and the process starts again.
You might be able to take some inspiration from it?
I've been using DataSync in prod for years, just looking at switching to Drupal Queue, but it doesn't offer the same immediate-ish asynchronous execution of tasks that DataSync can.
Comment #9
gordon commentedDrupal Queue is a direct backport of the Drupal 7.x Queuing system. Also it's interface is pluggable so you can use queuing systems like beanstalkd to do the heavy lifting.
If you use something like job_queue (not sure of data sync) then changing to Drupal Queues is a must.
Comment #10
neilnz commentedI know this issue is ancient, but just adding a little bit to it that I've worked on.
I've redone the patch a little to output the debugging prints to STDERR instead of STDOUT and to allow the pause between executions to be configured in a variable (drush vset only, no UI). Also applied the loop pause when repeating a certain number of times, as running a tight loop with --repeat=60 thrashes the database quite hard unnecessarily.