Best way to handle encoding on remote servers

Sounds very interesting. I'm wondering how this relates to remote stuff you can do with Media Mover. Isn't there something similar already? That might be a conceptually better place for it in the stack of modules, but I'm not sure.

Is ffmpeg_converter_remote available somewhere so people can take a look?

anything new on this?
subscribing

I've taken Brian's code and reworked it to make it as minimal as possible. For this to work the following patch will need to be applied to ffmpeg_wrapper.module. This allows us to use ffmpeg_wrapper_configuration_form as is, along with the mm_ffmpeg module. Otherwise, we'd have to copy all that code into a new module.

Conceptually what the attached ffmpeg_wrapper_remote module does is:

• Adds "FFmpeg Wrapper Remote Override" variable to ffmpeg_wrapper_admin form. Which toggles the calling the remote script instead of the local ffmpeg command.
• Adds "URL of remote script" variable to ffmpeg_wrapper_admin form.
• Adds "Path to files directory mounted on external server" variable to ffmpeg_wrapper_admin form. So the remote directory you mount your server files directory is replaced in the ffmpeg command sent to the remote encoder.
• Replaces $form['ffmpeg_wrapper']['ffmpeg_wrapper_vhooks']['ffmpeg_wrapper_vhook'] with it's own function call to get the available wrappers. I could have overridden ffmpeg_wrapper_vhook_list to the remote function. But, I was trying to make the ffmpeg_wrapper patch as minimal as possible. Therefore, the ffmpeg_wrapper_vhook_list() will be broken when using ffmpeg_wrapper_remote.
• Replaces the ffmpeg_wrapper_admin_validate with ffmpeg_wrapper_remote_admin_validate for the ffmpeg_wrapper_admin form.

? ffmpeg_wrapper.diff
Index: ffmpeg_wrapper.module
===================================================================
RCS file: /cvs/drupal-contrib/contributions/modules/ffmpeg_wrapper/ffmpeg_wrapper.module,v
retrieving revision 1.1.2.20.2.32
diff -u -r1.1.2.20.2.32 ffmpeg_wrapper.module
--- ffmpeg_wrapper.module	22 Nov 2009 20:39:07 -0000	1.1.2.20.2.32
+++ ffmpeg_wrapper.module	6 Dec 2009 21:39:40 -0000
@@ -248,6 +248,11 @@
  *   Output of the command.
  */
 function ffmpeg_wrapper_run_command($command = '', $error_check = true, $path = '', &$ffmpeg_object = null) {
+
+  if(variable_get('ffmpeg_wrapper_remote_override', FALSE) && module_exists('ffmpeg_wrapper_remote')) {
+    return ffmpeg_wrapper_remote_run_command($command, $error_check, $path);
+  }
+
   // override the system path?
   if (! $path) {
     $path = variable_get('ffmpeg_wrapper_path', '/usr/bin/ffmpeg');
@@ -977,6 +982,10 @@
  *   A full path.
  */
 function ffmpeg_wrapper_path_to_vhook($name) {
+  if(variable_get('ffmpeg_wrapper_remote_override', FALSE) && module_exists('ffmpeg_wrapper_remote')) {
+    return ffmpeg_wrapper_remote_path_to_vhook($name);
+  }
+
   if ($path = file_exists(variable_get('ffmpeg_wrapper_vhook', '/usr/local/lib/vhook/') . $name)) {
     return $path;
   }

ffmepg_wrapper_remote required encode.php running on the remote box to 'transfer' the command line instruction being sent to FFMPEG. What's required on the remote box with this change?

There is still a encode.php required on the remote box. Main thing is XML-RPC for PHP http://phpxmlrpc.sourceforge.net/ is required.

<?php

//
// Copy this script to a location on the remote server accessable via apache.
// Requires XML-RPC for PHP http://phpxmlrpc.sourceforge.net/
//

//Set this to only allow remote commands from a specific IP
$control_ip = '192.168.1.226';


include("xmlrpc.inc");
include("xmlrpcs.inc");
include("xmlrpc_wrappers.inc");

$request = $_REQUEST;
$user_ip = $_SERVER['REMOTE_ADDR'];

if($user_ip != $control_ip) { 
    return;  
}

$a=array(
  "encoder.runCommand" => array(
    "function" => "encoder_run_command",
    "signature" => array(array($xmlrpcArray, $xmlrpcString, $xmlrpcString,$xmlrpcInt,$xmlrpcString)),
    "docstring" => 'When passed a struct will run the ffmpeg command.' 
  ),
  "encoder.getVhookList" => array(
    "function" => "encoder_get_vhook_list",
    "signature" => array(array($xmlrpcArray, $xmlrpcString)),
    "docstring" => 'When passed a struct will run the ffmpeg command.' 
  ),
);
$s=new xmlrpc_server($a, false);
$s->service();

function my_exec($cmd)
{
  $proc=proc_open($cmd, array(1=>array('pipe', 'w'), 2=>array('pipe', 'w')), $pipes);
  $stdout=stream_get_contents($pipes[1]);fclose($pipes[1]);
  $stderr=stream_get_contents($pipes[2]);fclose($pipes[2]);
  $rtn=proc_close($proc);
  return array('stdout'=>$stdout,
    'stderr'=>$stderr,
    'return'=>$rtn
  );
} 

function encoder_run_command($m) {

  $p1= $m->getParam(0);
  $command = $p1->scalarval();
  $p2 = $m->getParam(1);
  $options = $p2->scalarval();
  $p3 = $m->getParam(2);
  $mm_fid = $p3->scalarval();
  $p4 = $m->getParam(3);
  $file = $p4->scalarval();

  if($mm_fid){
    //Run the command in the background

    $handle = fopen("/home/www/encodes/".$mmfid,"w");
    fwrite($handle,$options."\n");
    fwrite($handle,$file."\n");
    fclose($handle);

    return new xmlrpcresp(new xmlrpcval($handle, 'int'));
  }
  else {
    //Run the command and return
    $path = dirname($command);

    $output = my_exec($path.'/ffmpeg '.$options);

    return new xmlrpcresp(php_xmlrpc_encode($output));
  }
}

function encoder_get_vhook_list($m) {

  $m = $m->getParam(0);
  $path = $m->scalarval();

  $files = array();

  if ($dir = opendir($path)) {
    while (($file = readdir($dir)) !== false) {
      // do not this or parrent directory
      if ($file != "." && $file != "..") {
        $files[] = $path .'/'. $file;
      }
    }
    closedir($dir);
  }
  return new xmlrpcresp(php_xmlrpc_encode($files));
}

what's the update on this?

The way that services like encoding.com and flixcloud.com do this is that you send a XML request to their server that includes authentication information to fetch the file and return the file as well as the parameters that govern the encoding. Once the file is returned, they ping a URL on the production site.

In this case, I think you'd want the reverse. Have the remote machines ping production. If production has a file, return a XML request and lock the file until the file is returned.

I did a rough implementation for flixcloud and encoding which I'd be happy to share if somebody wants to adapt this. http://drupalcode.org/viewvc/drupal/contributions/modules/flixcloud_api/ is a much older version- I can share a more mature version

FYI, it would be *really* easy to pull something like this off with media mover which would handle all the file locking for you.

instead of using "paid per use" service, how about using local dedicated server doing just encoding work? this is a preffered option as there are deem to be many many video will need to be encoded.
so ffmpeg is actually running from another server?

Hi Arthur

I'd like to see your code. I modified mm_ffmpeg module to send a xml request to my remote encoder (the code I posted in #3) during the process step. The remote encoder then pings back to the server when done. I posted an issue on media mover to handle processing asynchronously to support this http://drupal.org/node/647082.

@najibx - sorry if I wasn't clear. The point that I'm trying to make is that the way the services work can be modeled by media mover.

@darrick - I'll commit my changes that allow the mm 1.x branch to do this, 2.x will handle this with grace. I'd love to help you port your code to 2.x if you're interested

@arthur - so the changes for mm 1.x branch will work for issue http://drupal.org/node/647082. The core of this issue is if the patch for ffmpeg_wrapper I mentioned in #3 can be applied.

In more words I have three things going on here:

- Request for media mover sub modules to be able set a custom status. Solved in http://drupal.org/node/647082.

- Request to apply the patch in #3 above to ffmpeg_wrapper. Otherwise I would have to duplicate a lot of code in the ffmpeg_wrapper_remote module.

- If the above is committed the two modules I have mm_ffmpeg_remote and ffmpeg_wrapper_remote will work with out any hacks. I would then commit them as separate projects but would prefer to work with you and have them committed as contrib modules for both the media mover and ffmpeg_wrapper projects.

I'm also interested in seeing the code you mentioned in #7 as I'd like to look into the possibility of abstracting my mm_ffmpeg_remote module to work with a number of different asynchronous offline encoders. Similar to how emfield handles a number of different providers.

Once this gets done I'm interested in porting to mm 2.x.

Ok, so I can commit the code in #3, but I'd prefer that this gets more abstracted- can we create variable setting which is something like "ffmpeg_backend" and this determines where the command gets processed? Sort of like how the mail system works?

Like this?

RCS file: /cvs/drupal-contrib/contributions/modules/ffmpeg_wrapper/ffmpeg_wrapper.module,v
retrieving revision 1.1.2.20.2.40
diff -u -B -b -r1.1.2.20.2.40 ffmpeg_wrapper.module
--- ffmpeg_wrapper.module	27 Feb 2010 22:18:40 -0000	1.1.2.20.2.40
+++ ffmpeg_wrapper.module	5 Mar 2010 01:02:03 -0000
@@ -248,6 +248,11 @@
  *   Output of the command.
  */
 function ffmpeg_wrapper_run_command($command = '', $error_check = true, $path = '', &$ffmpeg_object = null) {
+
+  if($ffmpeg_backend = variable_get('ffmpeg_backend', FALSE) && function_exists($ffmpeg_backend.'_run_command')){
+    return call_user_func($ffmpeg_backend.'_run_command',$command, $error_check, $path);
+  }
+
   // override the system path?
   if (! $path) {
     $path = variable_get('ffmpeg_wrapper_path', '/usr/bin/ffmpeg');
@@ -1038,6 +1043,9 @@
  *   A full path.
  */
 function ffmpeg_wrapper_path_to_vhook($name) {
+  if($ffmpeg_backend = variable_get('ffmpeg_backend', FALSE) && function_exists($ffmpeg_backend.'_path_to_vhook')){
+    return call_user_func($ffmpeg_backend.'_path_to_vhook',$name);
+  }
   $path = variable_get('ffmpeg_wrapper_vhook', '/usr/local/lib/vhook/') . $name;
   if (file_exists($path)) {
     return $path;

subscribing - thanks!

subscription! Wanna keep an eye on this one....

I've committed a fix for this on the 6.2x version. You just need to declare variable_set('ffmpeg_wrapper_engine', my_function_name) and function my_function_name($command, &$ffmpeg_object) {} you will need to return output from your engine. You can modify $ffmpeg_object->errors and what not.