Support multiple outputs per job for zencoder

This is absolutely true, and it was the way it worked before in the module for Drupal 6. Unfortunately, the current code structure makes it difficult to implement this quickly. I will look into it for a later version.

Here's a patch I've put together to add support for multiple Zencoder outputs per job. Would appreciate it if anyone interested would test it out and report back any success or problems.

[[UPDATE: HOLD OFF ON THIS ONE. I've found some further issues that need working out.]]

Ok, here we go. I've tested this against Zencoder with one and two presets.

It adds a new field to the video_output table, so be sure to run update.php. Let me know if you have any questions or feedback!

Thanks for the patch, I will look at it soon.

Tested the patch on a development version of a site I've got in production. In my context, I need to be able to add extra formats of the same video in order to have a truly mobile/responsive-friendly site.

Patch output follows:

patching file includes/Transcoder.inc
Hunk #1 succeeded at 166 (offset 50 lines).
Hunk #2 FAILED at 129.
Hunk #3 succeeded at 217 (offset 50 lines).
1 out of 3 hunks FAILED -- saving rejects to file includes/Transcoder.inc.rej
patching file transcoders/TranscoderAbstractionFactoryZencoder.inc
Hunk #6 succeeded at 233 (offset -5 lines).
Hunk #7 FAILED at 550.
Hunk #8 succeeded at 590 (offset 2 lines).
Hunk #9 FAILED at 611.
2 out of 9 hunks FAILED -- saving rejects to file transcoders/TranscoderAbstractionFactoryZencoder.inc.rej
patching file video.install
Hunk #1 succeeded at 188 (offset 7 lines).
Hunk #2 succeeded at 553 with fuzz 2 (offset 31 lines).

And failed patch produces this error:

PHP Fatal error:  Class 'TranscoderAbstractionFactoryZencoder.inc' not found in /home/www/XXXXXX/sites/all/modules/video/includes/Transcoder.inc on line 297, referer: http://dev.XXXXX.com/admin/config/media/video

What I'm not really understanding here is that it does appear that some of this work/idea has been brought into the module's code; I can, for example, request multiple files from Zencoder, and Zencoder processes them, posts them back, and the system does pull them in. But it does not add those additional video sources to the output.

Should this perhaps be branched off as a separate issue?

Tried the patch as well, and it failed. Probably needs a reroll against latest dev...?

Any chance of re-roll on that patch aasarava?

The question of what to do with multiple formats and/or variable bit rates is indeed a challenging one since they tackle different problems. A preset may need some additional context so we know what to do with the output once we get the file ready for playback. Thoughts anyone?

What I did once was to embed the bitrate (eg: 800k, 1200k) in the preset name. It will end up in the file name and in the player I made some modifications to pick the right one.

Which player were you using out of interest?

I used Video.js to play m3u8 files to stream to iOS.

I hope I'm not taking this off-topic, but I'd like to try and clarify a few things regarding what is meant by 'multiple outputs' from Zencoder.

As things stand with the latest dev version of Video module, multiple-outputs from Zencoder are supported - but they are treated as separate jobs. This does warrant a fix, and I'd be happy to test patch in #3 once it's been re-rolled to the latest dev release.

This isn't the whole story though, as twelvealpha points out in #5. As we move into the world of HTML5 and Responsive web apps that need to work across mobile we enter a whole new world of complexity. Not only do we require multiple-outputs - as in (mp4, ogv, webm, etc) to ensure we have compatibility within the spectrum of available browsers - but we also need multiple bitrate encodings of at least some of those formats, and the majority of us will be looking towards HLS and .m3u8 playlists to provide for IOS.

My first question re #10 is how did you manage to construct various .m3u8 outputs *and* and .m3u8 playlist when these options (as far as I can see) are not currently available within the presets module?

I know zencoder will happily provide everything we need from a single (decent resolution) source file, but we currently don't have any means to configure a video module preset/s to achieve this.

As I understand video module is now using Zencoder API v2 so all this is possible - at least in theory via a single Zencoder API request, but would it not require an overhaul of the presets module in addition to the Transcoder/TranscoderAbstractionFactory, some re-thinking of the db schema, and ultimately (for icing) integration with a least one suitable player (such as the latest HTML5 Flowplayer with ipadUrl plugin).

Understanding that this is no small feature request, given everything that is now possible using API v2 - a quick fix might be devise a feature that allowed a zencoder preset to be constructed manually as JSON/XML using a tokens for the source filename and various preset parameters? Or something to that effect....

What we need to get towards is typical API request for say a high and low bandwidth file, with HLS variants and a playlist....such as this:

{
  "input": "s3://zencodertesting/test.mov",
  "outputs": [
    {
      "label": "low",
      "format": "mp4",
      "video_bitrate": 200,
      "decoder_bitrate_cap": 300,
      "decoder_buffer_size": 1200,
      "audio_sample_rate": 44100,
      "height": "288",
      "url": "s3://example-bucket/low.mp4",
      "h264_reference_frames": 1,
      "forced_keyframe_rate": "0.1",
      "audio_bitrate": 56,
      "decimate": 2
    },
    {
      "label": "high",
      "format": "mp4",
      "video_bitrate": 1000,
      "decoder_bitrate_cap": 1500,
      "decoder_buffer_size": 6000,
      "audio_sample_rate": 44100,
      "height": "432",
      "url": "s3://example-bucket/high.mp4",
      "h264_reference_frames": "auto",
      "h264_profile": "main",
      "forced_keyframe_rate": "0.1",
      "audio_bitrate": 56
    },
    {
      "source": "low",
      "segment_video_snapshots": "true",
      "url": "s3://example-bucket/hls-audio-only/hls-audio-only.m3u8",
      "copy_audio": "true",
      "skip_video": "true",
      "label": "hls-audio-only",
      "type": "segmented",
      "format": "aac"
    },
    {
      "source": "low",
      "format": "ts",
      "copy_audio": "true",
      "copy_video": "true",
      "url": "s3://example-bucket/hls-low/hls-low.m3u8",
      "label": "hls-low",
      "type": "segmented"
    }
    {
      "source": "high",
      "format": "ts",
      "copy_audio": "true",
      "copy_video": "true",
      "url": "s3://example-bucket/hls-high/hls-high.m3u8",
      "label": "hls-high",
      "type": "segmented"
    }
    {
      "streams": [
         {
             "path": "hls-low/hls-low.m3u8",
             "bandwidth": 256
         },
         {
             "path": "hls-audio-only/hls-audio-only.m3u8",
             "bandwidth": 56
         },
         {
             "path": "hls-high/hls-high.m3u8",
             "bandwidth": 1056
         }
      ],
      "type": "playlist",
      "url": "s3://example-bucket/playlist.m3u8"
    }
  ]
}

For clarity on #11 - the really important thing here is that the HLS streams are not separate encoding jobs, they are 'segmented' versions of the previously encoded mp4 streams.

The resulting job creates 6 outputs - 1 charged at your account's regular rate, 1 with the 20% multiple-output discount, 3 transmuxed files at 1/4 of the encoding cost, and 1 playlist, which is free.

Once this side of things is solved, we'd then need to look at the player implementation - which is becoming equally complex - if not moreso.....

I'd be up for sponsoring if anyone is interested.....

Anyone still following this.....

Any chance of that patch against latest dev?

Did you have a chance to look at this patch Jorrit? If it looks good for inclusion in theory I can have it reworked against latest dev and will resubmitt...

I've been thinking about this feature request some more... and looking at the current code architecture.

Given the complexity of the API request I included in #11 as an example, and the scope of work required to update the presets and UI to cater for this level of complexity - could we not override the transcoder presets entirely and provide some kind of 'expert' mode?

1. Provide a checkbox in the Zencoder config (admin/config/media/video/transcoders) to override the presets module (and/or include this option in the video field settings).
2. Provide a field in Zencoder config that accepts a manually constructed API request using some variables for input and output URLs and paths. This data could potentially still be stored in the presets table as a BLOB as it's essentially a nested array of value pairs?

3. Modify TranscoderAbstractionFactoryZencoder.inc to use the expert settings as required.

OR is the work on the Presets doable using the following approach:

A. First add .m3u8 and 'ts' to the list of available formats/codecs (simple).
B. Specify a Presets input file to be the output file of another preset rather than the original source file (simplish).
C. Add a checkbox option to include a specific Preset output as part of an overall .m3u8 stream, and a field to specify the required playback bandwidth for the stream (or could the target encoding bandwidth be used?).
D. Modify TranscoderAbstractionFactoryZencoder.inc as required.

Questions:

Does this cover everything on the encoding side? What else would be need to be done - aside from display formatting?
Would a new module that interfaces on with Video be a better way?
Should Presets really be catering for both Zencoder and FFMpeg at the same time?

Thoughts anyone?

You are right that it is not a good idea that FFmpeg and Zencoder use the same preset form. In Drupal 6 this was different.

Multiple outputs per job should in my opinion be implemented by removing the setInput, setOutput, setOptions, execute calls and replace them with a method that executes for multiple presets. Again, this is how this was done in Drupal 6.

In any case, I want to make a new release soon and start working with large changes like these only after that.

Any movement in this area?

Cheers

None, I'm afraid to admit.

Yes, this would be greatly appreciated! Then I would be able to serve smaller videos to mobile phones.

is there any movement on this.
I am thinking to overrride whole process and inject manually created API call to generate multiple output in one call.

I am also planning to not play too much with postback function. For this, I assume on playback that xxx formats and bitrates are available for transcoded file. This way i dont need to save all info about file in database.

Let me know if you have better idea to serve this. thnx