Avoid race conditions at maestro_orchestrator()

I took a quick look at this patch -- looks good overall and uses the semaphore approach I noted in another issue thread. Nice job.

I still wonder about the database based semaphore approach with respect to injecting race conditions on busy sites. If the database transaction does not block or is incapable of blocking another request, then this approach may also fail. I also have to check in to the ability of the Drupal lock to be unlocked after a certain time. If the unlock fails at some point prior to the lock condition being cleared, the orchestrator will appear hung.

This looks good Aron.

However I see a few things that I may tweak before committing this:

1. add the lock delay as a config parameter for Maestro.

2. add the ability for cleanqueue to ask to run, be blocked (by the lock), and go in to a wait state for a configurable amount of time and then try to re-run.

3. add a configuration option to denote how many times to do #2.

I also see a more user friendly interface where if #2 happens and is blocked, that a redirect happens to a page that tells the user something friendly and refresh the appropriate number of times denoted by #3. When done, redirect to the task console.

I'm not sold on the last section above yet.. but it's an idea.

Thanks for this Aron. I used your patch. It is a good start in to allowing the orchestrator to at least run on demand for edge cases.

Hey guys, just checking to see if there was any progress made on getting this into Maestro core. I have Maestro running as the backbone to a web application I created and we have the ->clearQueue method running at several points in the app to move things along in "real time." So far we have not noticed any ill effects with this, but would be interested in the above to make sure.

@ryan.armstrong The locking mechanism has been put in place in Maestro for quite some time. Latest full release has it in there as well.

Changing the status to closed.