Per-preset parallelized engine processing [#1323430]

I'm not experienced, but the goal is to make parallel request to the engine script, wich becomes independant.
Maybe the solution is here : HTTP Parallel Request Library
Thus media_derivatives_start_encode() call the engine script in a specific file (media_derivatives_start_encode.php) with a non-blocking http_request, and sends the derivative object or any information required. This script creates a database entry with process status, then starts transfer/encoding/creation of derivative. Once completed, the script calls media_derivatives_derivative_finished() and update the database entry.
I've never used functions like that and i don't know if it's the right way, but this issue is here to discuss this subject.

Comment	File	Size	Author
#12	1323430_set_progress_information_12.patch	1023 bytes	slashrsm
#4	1323430_background_process_4.patch	3.16 KB	slashrsm
#3	background_process-1323430.patch	593 bytes	camdarley

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

slashrsm CreditAttribution: slashrsm commented 31 October 2011 at 18:55

I took a look at HTTP Parallel Request Library and I do not exactly see how to use it for this use case. media_derivatives itself do not call any http requests, so we could not use this module.

I also took a look at core's Batch API, but as I understand we cannot use it without default progress page. I think we do not need this progres, since we want to run processes in background, without any UI.

Searching further I found Background Process, which allows us to run processes in background. Processes can be accessed later, so we could build a UI list of currently active processes. It think this could be implemented as a recommended only dependency. media_derivatives could check if this module exists and use it. Current behaviour could be kept if there will be this module missing. Do you think this is right behaviour or should be make it a dependency?

Comment #2

camdarley CreditAttribution: camdarley commented 2 November 2011 at 13:57

You're right, i didn't know about Background Process but it seems to be exactly what we need.
You're right too about implement it as a recommended only dependency, although, in my opinion, all the engines take a long time to generate their derivatives. But as Background Process has its own dependencies, it may do a lot of modules to install, and it's better to let it optional.
I think the best would be creating a scheduler: 'Runs derivation in a background process' (with a modification of media_derivatives_schedule_derivative() to handle it) which is only accessible when Background Process is installed.

Comment #3

camdarley CreditAttribution: camdarley commented 24 November 2011 at 14:06

Status:

Active

» Needs review

File	Size
background_process-1323430.patch	593 bytes

I made a patch to implement Background Process.
I made a few test and it's working.
Maybe it could be better to have an option to activate or not...

Comment #4

slashrsm CreditAttribution: slashrsm commented 27 November 2011 at 22:56

File	Size
1323430_background_process_4.patch	3.16 KB

I added an option to enable/disable this. Can you check it a bit?

Comment #5

camdarley CreditAttribution: camdarley commented 28 November 2011 at 10:14

Status:

Needs review

» Reviewed & tested by the community

Patch work fine for me. Thanks. Ready to be committed.
Background process provide a page to see all jobs processing. I'll dive into the code to see if it's possible to have a "derivatives only" similar page.
Also in the future, it might be useful to have a per preset option... what do you think about this?

Comment #6

slashrsm CreditAttribution: slashrsm commented 28 November 2011 at 12:07

That sounds reasonable. I could try to implement this in the following days and we could commit it together with this patch.

What do you think?

Comment #7

camdarley CreditAttribution: camdarley commented 28 November 2011 at 13:18

I agree.
About the derivatives' background processes' page, here are my first thought:

Processes are identified by a md5 key. In my opinion, we should associate this key with the derivative being encoded, either with a join table, or with a new field in media_derivatives table. I do not know which solution is best/easiest to implement. With this association we could be able to list all derivatives processes.
A second thing is the progress indicator, which is implemented in Background Process. The great improvement would be the ability to monitor the derivative encoding progress. But we need two things for that.
1. Create a new hook for engines: hook_media_derivatives_get_progress($file), which return a percent value. For example, Youtube API allow to retrieve the current uploading state, ffmpeg allow to write a log file with encoding progress, etc...
2. I do not really understand how the Progress module works. But we must find a way to influence the progress indicator function. The advantage here is that Background Process seems to have everything we need to have a real-time javascript-based monitoring. For now, it just switch from 0% to 100% when finished.

If the first point could be easily committed, the second one need more investigation to find the right way.
I'm not actually a developer. I try to do my best but I lack some concepts of code.

Comment #8

slashrsm CreditAttribution: slashrsm commented 26 January 2012 at 22:20

Title:	Parallelize engine processing	» Per-preset parallelized engine processing
Status:	Reviewed & tested by the community	» Needs work

Commited: http://drupalcode.org/project/media_derivatives.git/commit/fad70fd

Thanks for that. I'm sorry because it took so long.

Renaming this issue to indicate what still needs to be done (comment #5).

Comment #9

gielfeldt CreditAttribution: gielfeldt commented 13 February 2012 at 21:06

To update the progress, this must be done from within the background process. Somewhere inside media_derivatives_start_encode() or one of its sub-functions you can do something like this:

if (module_exists('background_process') && $handle = background_process_current_handle()) {
  $message = "Processing this and that";
  $progress = <progress ranging from 0.0 to 1.0>
  progress_set_progress($handle, $message, $progress);
}

Comment #10

slashrsm CreditAttribution: slashrsm commented 14 February 2012 at 08:21

I think this should be done in encoding function itself, as those can determine, if even possible, how much work has been done.

Derivatives could implement API function that would wrap your code. Encoding function would then just call something like this:


media_derivatives_set_progress(<progress ranging from 0.0 to 1.0>)

Comment #11

gielfeldt CreditAttribution: gielfeldt commented 14 February 2012 at 08:43

Sounds reasonable. I was also thinking of creating a background_process_set_progress() function to take care of the handle check, but that might be overkill/unnecessary.