Posted by jhodgdon on February 26, 2013 at 7:07pm
2 followers
| Project: | Drupal.org infrastructure |
| Component: | api.drupal.org |
| Category: | task |
| Priority: | normal |
| Assigned: | jhodgdon |
| Status: | needs review |
Issue Summary
Occasionally, updates to the API module cause all of the API files to need reparsing on api.drupal.org (and on the associated staging site).
As things are now, we have to wait for quite a lot of cron and queue-cron Jenkins jobs to run, in order to see if (a) the parsing succeeded without errors and (b) the site is working correctly after all the files have parsed.
So, what we need is a new Jenkins job that we can launch on demand that will parse larger chunks of the files.
We have a sample script at:
http://drupal.org/node/1516558
(bottom of the page). It needs to be adapted to work via Jenkins -- something like what the live/kick-views-cache.sh script in the Infra git repo does.
I'm working on it... will attach a patch shortly (hopefully).
Comments
#1
OK, I have a script that seems to run fine from the command line on stagingwww.drupal.org.
What I did, in my home directory there:
- Cloned the infra repo
- Added this script (api-parse-batch.sh) to the "staging" directory (see attached patch).
- Ran this command to set an environment variable apparently needed by the common.sh script:
export JOB_NAME="api.staging.devdrupal.org"
Note: I thought the job name should be something like "api.staging.devdrupal.org something something something" but this did not work in the common.sh script?
- With this environment variable set, from the root of the infra git repo, ran:
staging/api-parse-batch.sh
The script ran fine. I had temporarily set it up to run just 2 branch update, 2 queue-cron, and 2 node cleanup jobs, which finished fairly quickly.
The one thing I noticed is that the -v option for drush is not on by default, meaning that only errors and warnings are shown from watchdog. I'm not sure whether this makes sense or not -- there would be a LOT of output if we used -v (all of the messages saying which files were being parsed). Thoughts?
Anyway, here's the patch.
#2
http://drupal.org/project/queue_runner exists as of January.
#3
I don't see how Queue Runner helps at all. It still uses Drupal core cron jobs to run items in the queue, and cron jobs will already be running items in the queue. This issue is about setting up a Jenkins job to script running a bunch of parse jobs at a time, rather than waiting for cron to clear out the queue as it would eventually do normally.