This is probably somewhat beyond the intended scope of the module... but then I cannot think of any other module that could benefit more...

Since we are dealing with moving/processing media files, distributed processing comes to mind (as has been mentioned in the discussion already) and the need for a job/message queue becomes more evident then ever. (The existing "job queue" module is just a baby step in the right direction.)

At minimum, Media Mover could be designed to OPTIONALLY rely on an external message/job queue. The fact of defining such an API should go a long way towards inspiring others to implement the queuing mechanisms.

I foresee that we will end up with at least four queuing options (beside "none"):
- current basic job queue module
- Zend job queue
- some kick-ass Open Source messaging queue
- Amazon's new queuing service

Granted, at the end of the day the job/message queue API (especially for distributed media processing) belongs in the core... But Media Mover is clearly THE best starting point for the effort.

Comments

arthurf’s picture

arthurf commented

This is an interesting idea. I don't see it as being overly complicated to implement- there is really one function where this would be tied in (media_mover_api_run_config() ) where messages could easily be passed out. Probably what ought to happen in doing such a thing would be changing over from a "one file at a time" system to a "one process at a time" system, so you could actually make use of a que system better.

If you'd like to get involved in trying to work on such a system, I'm totally behind it. The way that media mover handles messaging right now is a bit cumbersome- I'm working on improvements, but I've got other obligations as well. At anyrate, please keep this converstation going, I'd love to see what kinds of implementations it could have and again would be happy to help steer the code in this direction.

andrabr’s picture

andrabr commented

I am new to Drupal, so I will probably get up to speed enough to contribute by Feb., but meanwhile here is a collection of relevant links for the interested to ponder:

Other queuing efforts:

  • Simple Asynchronous Messaging for PHP - seems to rely heavily on IBM backends
  • There are multiple Java queuing projects (a lot of them half-baked)
  • AMQ - a GPL'ed messaging effort pioneered by the JPMorgan (very good intentions but Wall Street people seem to be clueless about what makes an Open Source project truly flourish)
  • Active MQ by Apache Foundation - actually has PHP clients (personally, I find the idea of a Java-based message broker as inappropriate as a Java-based database engine)

Sadly, the queuing world is in much more disarray and confusion than Web server world: lots of "emerging" protocols and often poor PHP client support.