In a nutshell, this will introduce rich media display within Drupal core using the <video> and <audio> HTML5 elements. Initial development will occur at http://drupal.org/project/html5_media and will then migrate to a core patch in this issue thread.

To read a full description of this initiative along with roadmaps, etc. Please read the following.

Thanks,

Travis.

Files: 
CommentFileSizeAuthor
#39 drupal-video_audio_support-1174892-39.patch368.66 KBtravist
PASSED: [[SimpleTest]]: [MySQL] 35,690 pass(es).
[ View ]
#22 drupal-video_audio_support-1174892-22.patch347.06 KBtravist
PASSED: [[SimpleTest]]: [MySQL] 34,811 pass(es).
[ View ]
#21 drupal-video_audio_support-1174892-21.patch1.42 MBtravist
PASSED: [[SimpleTest]]: [MySQL] 34,794 pass(es).
[ View ]
#17 drupal-video_audio_support-1174892-17.patch1.4 MBtravist
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal-video_audio_support-1174892-17.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#19 drupal-video_audio_support-1174892-19.patch1.39 MBtravist
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal-video_audio_support-1174892-19.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#1 1174892-html5-audio-video-tags.patch1.61 KBDave Reid
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1174892-html5-audio-video-tags.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Comments

Status:Active» Needs work
StatusFileSize
new1.61 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1174892-html5-audio-video-tags.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Very basic initial patch.

Todos:
- Make actual theme functions for audio and video.
- Support the audio/video element in the formatter settings.

Dave,

This is definitely a great start, but I do see one issue with this approach. I can see you are using the multiple file upload to print out a new player for each file uploaded to the file field, so that the patch above would produce the following output...

  <video src="file1.mp4"></video>
  <video src="file2.mp4"></video>

IMO, we should use the multiple upload capabilities from file field to provide the ability for the user to upload several formats of the same video ( OGG and MP4 ) so that it will work as HTML5 for most modern browsers. The code for this is actually something like this...

  <video>
     <source src="file1.mp4" type="video/mp4">
     <source src="file2.ogv" type="video/ogg">
  </video>

I already have this working in the HTML5_media module above so check that out and let me know what you think, and of course, please give me your thoughts on this approach.

Travis.

I like your approach just using the formatters and not a widget for media display... I will be making those changes to the html5_media module, and then make some changes to include the source element if you agree with my comment above.

Thanks again.

Travis.

Subscribe! :D

Hmm, when supporting multiple src we're going to want to figure out how core is going to handle one element with multiple attributes of the same type.

We need to figure out how we're go to account for one video element having multiple srcs, examples:

<?php
array(
'type' => 'video',
'src' => array(
 
'video/mp4' => 'file1.mp4',
 
'video/ogg' => 'file2.mp4',
  )
)
?>

The above example should print

<video>
     <source src="file1.mp4" type="video/mp4">
     <source src="file2.ogv" type="video/ogg">
  </video>

Yet

<?php
array(
'type' => 'video',
'src' => array(
 
'video/mp4' => 'file1.mp4',
  )
)
?>

should print
<video src="file1.mp4"></video>

This is just me subscribing to the issue ;)

I believe that it should be up to contrib as long as it's possible to do. Multiple-source video fields could use Field collection and a small contrib module could be written (or put in html5_media) for the multi-source video field collection formatter.

I think the 80% of use case will just want to upload a video, not have multiple formats. We should be working to do that in core, but ensure the more complex use case is possible in contrib.

Thinking more about this, we may have an issue we need to overcome...

Dave, I agree with your 80% as long as they have a Flash/Silverlight fallback in place to allow that single video to be playable in all browsers. Herein lies the problem. In order to do Flash fallback the right way, you still need to nest the object element within the video element as follows..

<video src="myfile.m4v">
  <object .....   />
</video>

If we are to go ahead and build this nesting to accomplish fallback, then my question is can we build this to work for the source element as well? What are your thoughts on that? Also, is there currently a way in Drupal infrastructure to get this type of element nesting on a per-field basis?

Your help and guidance is appreciated.

Thanks,

Travis.

MediaElement does have Flash and Silverlight fallback included in the library.

The problem with HTML5 video fallback is that it doesn't work if you are using a user agent that supports HTML5 video.

That is if you are using Firefox 8 and embed an H.264 video as the src of the video Firefox will not play the video, because it doesn't support H.264, but it also will not render the fallback content, because it supports the video element.

See https://developer.mozilla.org/en/Using_audio_and_video_in_Firefox

I understand that MediaElement has Flash fallback, but the biggest problem I see is that it writes HTML output dynamically using JavaScript. This will completely undermine any theming efforts where theme developers would like to alter the markup generated by this library. The better option is to allow the themers decide how the markup of their player should look by using the Drupal theme layer (maybe a media_player.tpl.php?) which will allow them to create their own themes for their players. I envision future themes would have a media player that fits perfectly in with their theme.

Another concern I have with MediaElement is it is huge. Last time I checked the download was 23mb. Understand that we could remove some of the libs he includes, but it would still add a significant footprint to drupal core. My goal is to create the absolute minimalistic player with a plugin type architecture that would allow contrib modules to add functionality to the player. All of this is already working within the http://drupal.org/project/html5_media project.... I just need to go ahead and create a patch in this issue with that project ported to the latest D8 branch, but please take a look at that project files and "player" folder to see what I am working on. I am not saying that this is the absolute right way to go, but I would at least like to have people give it a test drive and see if it would work for Drupal core.

There are some things that need to be figured out with what I did, like uploading additional files like subtitles to be shown with the media, but I think a first run patch is needed. I will try to have a patch within the next two weeks.

Component:file.module» file system
Status:Needs review» Needs work
Issue tags:-Media Initiative

Here is a github repo of what I am working on. It is easier to explore the code.

https://github.com/travist/drupal_multimedia

Component:file system» file.module

Sure the actual source files of Mediaelement that would be used as a library are 430KB which includes the silverlight and flash files. I understand the concern about theming, but for most people that just want to upload a video and play it, they likely would be overriding a simple formatter anyway if they want to theme it.

I'll be sure to take a look at what's going in the Github repo for drupal-html5-player! My concern with rolling our own library is just frankly core code maintenance. Once we put it in we have to maintain it forever. That's not a decision to take lightly.

Yeah, I am officially putting my foot in my mouth because looking at his library, I see that 20 of those MB's are included media. So, yeah that argument is invalid. :)

But I also want to mention another concern. That is that MediaElement.js cannot use some libraries that are already included in D7-D8... primarily jQuery UI. Take a look at this code... https://github.com/johndyer/mediaelement/blob/master/src/js/mep-feature-.... Here he is replicating code that jQuery UI already figured out with their slider control. Here is what that code would look like using jQuery UI ( https://github.com/travist/drupal_media_player/blob/8.x/core/modules/fil... ).

I think my point is that we have the ability to make a Drupal specific player that will completely cater to the themeing community to where they would not have to alter it, but rather build on top of it and also provide the markup they want in their theme. This would be very powerful. If you think that people will just override something, then we probably should keep it out of core all together. The goal here is to try to attempt to get this much needed functionality into core.

Your thoughts?

If the module is using any jQuery UI components, and you would like it to be considered for Core, then one important step is to make sure that the implementation of the components used doesn't have any accessibility problems.

You can see a list of open accessibility bugs against jQuery UI at http://bugs.jqueryui.com/query?status=!closed&keywords=~a11y

Issue tags:+Media Initiative

Status:Needs work» Needs review
StatusFileSize
new1.4 MB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal-video_audio_support-1174892-17.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

So, I know there will be some issue queue churn from this, but I want to get this out there so that everyone has time to digest before we meet in Denver to discuss this.

Here is my patch that introduces <video> and <audio> support into Drupal 8! Instead of going into detail of what this patch provides, I recorded a video where I walk through what this introduces into Drupal core.

http://www.youtube.com/watch?v=qY8ezHNiZPA

In addition, I have committed my changes to https://github.com/travist/drupal_multimedia, and you can view the changes made to core by looking at the following URL.

https://github.com/travist/drupal_multimedia/compare/8.x...8.x-media

If anything, this is a good solid start to get this functionality into Drupal 8.

Comments and Critiques welcome!

Travis.

The last submitted patch, drupal-video_audio_support-1174892-17.patch, failed testing.

Status:Needs work» Needs review
StatusFileSize
new1.39 MB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal-video_audio_support-1174892-19.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Let's try this one...

Status:Needs review» Needs work

The last submitted patch, drupal-video_audio_support-1174892-19.patch, failed testing.

StatusFileSize
new1.42 MB
PASSED: [[SimpleTest]]: [MySQL] 34,794 pass(es).
[ View ]

Alright, this one applies cleanly on my sandbox. Didn't realize that you had to use git diff --full-index --binary

Component:file system» file.module
Status:Needs work» Needs review
Issue tags:+Media Initiative
StatusFileSize
new347.06 KB
PASSED: [[SimpleTest]]: [MySQL] 34,811 pass(es).
[ View ]

So, the previous patch was pretty huge because I included the automated documentation which isn't necessary. Here is a new patch with only the required additions necessary to get this working (without the documentation for the player).

I have to agree here - the idea of using a template file and being able to build and theme the player the way I'd want it is extremely powerful. Overriding formatters is hard :)

Issue tags:+sprint

Some community reviews and feedback would be great to see here. Tagging for the current sprint.

Ok my feedback / own opinion

1) You should wrap all the video/audio stuff in a media field somehow. It will be confusing to use file field for this and text field for that.
2) Don't (as in rly don't) use a table in field formatter settings. Use plain text seperated with a
.
3) The media player is a seperate non drupal library, right? So we should place it inside misc (like farbtastic)

Could you split the player from the drupal code when you reroll. Makes it easier to review

I am totally open for feedback here, and was actually thinking that the player could live in the misc folder. So, for that reason, I don't see any problem with putting the player within the misc folder, but would like to get other opinions before I do a re-roll with this. Also, I will be in Denver and would like to meetup with everyone to go over this and we can talk it through there.

I would be ok with having a default media player in the misc folder as long as it is easy to override that player with another one. Custom video players are a frequent request / requirement of clients. Often times the video content network they pick governs the kind of player that can be used as custom branding of the player was built in (typically flash based players)

That is another cool thing about this player is that it is really easy to build shims around those content video networks custom players and have it plug right into this framework providing a common API between it and say regular HTML5 media control. I have already written one for Limelight CDN (who has their own player), but using their JavaScript API you can create the shim I am referring too. Regardless, this media player is only brought in if they select the "Media Player" field formatter, so it can easily be swapped out for something else if you just provide a different field formatter.

I just saw the video and this looks really awesome! I wish things were that simple in D7 too.

I agree with Bram's 3rd point in #25, it is somewhat of a UX WTF to have to figure out/remember that you need to paste the URL in a text field that is then converted to a media player on display. Users enter a URL, so they expect the field to be at the very least titled so (even if in fact it is nothing more than a text field). How about using a "special" kind of text field called "Media URL"? That would perhaps also allow us to have it have a media player widget assigned for its display by default. Don't you thing that this would be more streamlined?

We now have form api support for url fields. Why not use that?

I haven't looked at this issue in a while, so forgive me if any of this has been addressed.

The UI of the media player will need to be assessed for accessibility. High level:
- Can it be used with keyboard alone
- Can it be used with a screen-reader
- Does it have resizable text and proper colour contrast

For video, what formats does the player support, if any of those formats support caption/subtitlinghow does the player handle that?

For audio / video, there should be a field available to add a URL to the transcript, or to paste the transcript.

Everett,

Currently the player does respond to the typical keyboard inputs including Space for Pause/Play, Right and Left for seek forward and backward, and of course ESC to exit full screen mode.

As for subtitles, that could be added without to much hassle since HTML5 provides a <track> tag that provides subtitles within the HTML5 player. I plan on implementing that really soon and pushing up a fix for that so I don't think that will be an issue.

As for resizable text and proper color contrast, this is controlled via CSS so I am confident that this isn't an issue as well, and if it is can easily be fixed.

In the next day or so, I will be pushing up a new version of the patch with an updated player code in it (which provides true fullscreen support for supporting browsers), and when that happens I would love for you to perform an accessibility audit on the player to make sure it upholds all your team requires.

Thanks,

Travis.

@travist

Thanks for the feedback, sounds good.

Can you please provide a demo URL when you push your changes, this makes it far easier for others to provide quick feedback.

I just created a BOF in Denver for us to get together and go over this...

http://denver2012.drupal.org/bof/rich-media-handling-and-drupal-8

So, I think we should embrace this as the "core" player for Drupal.

http://popcornjs.org

This is a well established "core" player as I described above, but is maintained by the Mozilla Foundation. I do think that we will need to add some stuff to it to get the theming support, but hopefully this should be a really good start. I will start working on integrating this instead of the minPlayer and try to update this pull request accordingly.

Exciting stuff!

Travis.

I'd love to see this as a module for D7 too! Even if it is not accepted for D8 core, it's worth having it in contrib.

This is already in a D7 module, which is http://drupal.org/project/html5_media. However, I am going to make a patch to the Media module on Friday to basically move this into the media module instead, which would end up obsoleting the html5_media module.

Absolutely on the exciting...

Would really like to know what the ramifications would be of adding this.

StatusFileSize
new368.66 KB
PASSED: [[SimpleTest]]: [MySQL] 35,690 pass(es).
[ View ]

Here is the latest patch for this. I have still not had time to look into popcorn.js as the "core" for this media integration, but this is at least a good foundation to experiment with the integration with Drupal.

Couple points:

In response to #6 and #8 (which are a year old, so maybe opinions have changed): I can't think of a reason why anyone would be interested in adding a single format. Even if they do want to, it's the equivalent of (ok, it just is) coding an i.e. only site, and we certainly shouldn't be encouraging it. From a ux perspective, most end users will probably only want to upload one file, but that's what projects like Derivatives API and Media Derivatives are for.

JS libraries: Popcorn seems like overkill just for playing files -- from what I see its main use case is tying time based events together inside and outside an HTML media element. One of the major benefits of media in HTML is that the browsers supply controls natively. It seems to me there's no need for any js (even for fallback if we use Video for Everybody, below).

Structure: The most robust video tag structure seems to be Video for Everybody. It's well tested and as a recommendation on http://html5please.us has high visibility and consistent refinement.

We could use VfE syntax with no js whatsoever. If we want to go crazy with validation, we could make things extra light for modern browsers by removing the flash fallback in html and feature detecting html5 media element support (with modernizr and modernizr load aka yep/nope, or whatever feature detection and conditional loading is available in core), doing nothing if it passes and loading mediaelement.js only if it fails. Light and fast HTML5 media for supporting browsers, polyfill to add support for those that don't.

One of the important themes I see throughout initiatives for Drupal 8 is keeping the front end simple, allowing developers build up as opposed to forcing them to strip down. Simple, functional support for the elements should be the goal for core. If theming the player's controls requires an extra js file (mediaelement.js) let's make it optional, or better yet a contrib module.

[edited and expanded]

Hello Rob,

I appreciate your input on this and am more than intrigued at this approach. My only concern is with regards to consistency, which is something that Video for Everybody does not address at all. It primarily rely's on the native controls which is inconsistent across ALL browsers. I am not necessarily saying that this is a problem and some people probably don't care, but there are plenty that do care about this issue. Considering that Drupal aims to provide a consistent user interface, I would think that this is something that should be addressed within core. But you are right... to provide consistency requires, and is heavily dependent on Javascript. There are goods and bads either way, and I would certainly love to hear what everyone else thinks about this.

I am pretty slammed at the moment, but once things calm down, I may prototype this up in code and just see how it compares to the previous solution.

Thanks again for your input,

Travis.

Hey Travis, thanks for the reply.

I spent some time looking through the HTML5 Media code today, and I think I understand more what you're looking to accomplish. It seems like the project's main goal is a consistent theming api for HTML5 audio and video tags, which I can completely get behind. In order for that to work there needs to be a way of adding, formatting, and theming the tags themselves, which you've provided, but only coupled with the first use case.

I think this could be a complete core solution if it moved in two steps:

  1. Provide formatters that produce the naked HTML5 audio and video elements.
  2. Extend those with the minPlayer for complete, Drupal style theming control.

One of the points brought up in the convert Drupal 8 theming to twig discussion was that Drupal can benefit from being less proprietary and more accommodating to front end developers who have their own toolsets and techniques that they apply across any number of projects. A two step system would cover developers who want the lightness of native code, those who have a HTML5 media player that they prefer from use across all of their projects, and those who like minPlayer because it works out of the box and does things the Drupal way.

Speaking to consistency, native players are all different, but they're also more performant and offer a consistant experience in browser. Think check boxes and radio buttons -- it's not considered bad form (HA groan) to let browsers implement native styling there, but if you want to put the extra effort into images, css, and some wrapper tricks, a custom style can better compliment design. I think video and audio are pretty much the same. There's clear benefits to both approaches.

Anyways, thanks for all the work you've put into this module so far. Testing and experimenting with it in a side project as we speak.

Status:Needs review» Needs work

I applied the patch from #39. It installed correctly and when I created a file field with an .mp4 video file uploaded, the manage display setting on the field was 'media player'; the video did not display. In the 'recent log messages it had 'Theme key "media_player" not found.'

Version:8.x-dev» 9.x-dev

Drupal 9 at this point. We did add support for HTML5 audio and video in the File entity module.