Incorrect Amazon S3 Private URL Passed to Flowplayer 3 after updating (CORRECTION)

Subscribing. Any progress on making a fix official?

@ copeasetic Were you able to get Private (signed) URLS to work between the video module and your S3 bucket? If, so was it just that fix? I make the change above, yet when I enable no future uploads work.

Thanks!

- Josh

What kind of encoder are you using or are you using no transcoding?

Question:

Does the video module work with private URLs and Zencoder transcoding? If so, what version are you running?

When enabled, should it be just a re-upload of the videos?

Thank you in advance.. been banging my head on this one..

- Josh

just for info: I have the same problem, it does not work with any version. URL break.

I'm waiting on new information until I can use the modules. Am on any news I am grateful.

regards

I have the same problem. When I use "Enable Private" under Filesystem tab, Zencoder fails to upload video to Amazon S3. I'm using Video 6.x-4.5.

Is it possible that Zencoder does not have permission to write to your bucket?

Are there any errors logged in Drupal or Zencoder?

Yes, Zencoder is getting permission denied when uploading to S3 bucket but I don't have this problem when I don't enable Private URL. Zencoder has read-write access to that bucket.

### Error log in Zencoder
Access to the file at 'https://mybucket????/files/videos/video_tutorials/1/1111/test_2.avi' was denied (HTTP 403 response code). If this is a S3 URL, you probably need to grant read access to Zencoder for this bucket or file. Get more information in the documentation.
###

Do I need to make changes to S3 ACL when Private URL is enabled?

I am currently busy refactoring the S3 / Zencoder code an I will address this issue as well. I will release a beta version of 4.6 soon. I hope you are willing to invest some time to help test that release.

Thanks Jorrit,

I can definitely help out to test the release with Amazon S3 and Zencoder.

Please try if version 6.x-4.6 beta 1 fixes this problem for you. In this version the Zencoder and S3 parts have been refactored and tested.

You can read about it at http://drupal.org/node/1346434

Great job, It's totally fine now and Zencoder has no problem uploading to Amazon s3. Very appreciate your prompt response and fix. Any idea when the production version will be ready? I noticed you have some debugging enabled for this Beta version.

Still no-go for me.

I was running 4.5, so I disabled all Video module components, deleted from modules directory, then uploaded the beta, activated, and updated the DB.

I promptly enabled "Private" files and made sure that the bucket was private as well (my user account and Zencoder have GET privileges, among other capabilities), deleting my "files" directory, knowing it will be rebuilt upon file upload.

The Drupal -> S3 -> Zencoder -> S3 -> Drupal (Postback) all works perfectly- as it has in v 4.5.

However, the page shows a big blank box where the player should be. Using FireBug, here is what the URL looks like (edited):

"https://bucketname.s3.amazonaws.com/sites/default/files/converted/videof... %3D"

Note that it looks like a space and a = is attempting to be encoded at the end of the signature.. is that correct?

I am using SWF Tools 3.0 Beta and have tested with JW Player and FlowPlayer with the same result.

If I were to disable "Private" in the Video / Filesystem tab, make sure the bucket is public again, then upload a new clip- it all works swimmingly.

Any ideas what might be different for me? Thank you in advance for your help.

First of all, thanks for testing the new release.

naderat: Could you tell me what debug information you are getting? Is it debug messages in the Drupal log file? This version will be made final when it has been tested by the community. As you can see, there may be still some problems with it.

jhrizz: That URL looks strange indeed. Are you using Flowplayer using SWF Tools or via the Flowplayer API module?

Here are some things you can try:

• Check if your system clock is set right. If it's not set right, the URL is expired before it is valid.
• Increase the private URL lifetime by a large amount (just as a test).
• Try the built in HTML 5 player.
• Immediately after you display the broken player, locate the video URL using firebug or viewing the source and open the file in your browser. Post back the XML you get, if you get any. Also please post the time and time zone you're in.

I have tested this myself now and my conclusion is that SWFTools and/or Flowplayer have difficulties with ampersands in URLs. If you can get Flowplayer to work using the Flowplayer API module, that would be the best solution in my opinion. I could try to supply a patch to Flowplayer, but the fact that they have a beta version outstanding since March and 140 open bugs suggests that the module is (almost) abandoned.

* Check if your system clock is set right. If it's not set right, the
URL is expired before it is valid.

- Dev on BlueHost, not sure how to check, other than all DateTime functions work fine.

* Increase the private URL lifetime by a large amount (just as a test).

- Did, no dice

* Try the built in HTML 5 player.

- Did, no dice

* Immediately after you display the broken player, locate the video URL
using firebug or viewing the source and open the file in your browser.
Post back the XML you get, if you get any. Also please post the time and
time zone you're in.

- Did, and it played!

Also switched to the build in player, and I get an X and no video.

I checked the logs, here is what I found...

Details
Type	filefield
Date	Saturday, November 19, 2011 - 05:51
User	admin
Location	http://www.sitename.tv/vod/test
Referrer	http://www.sitename.tv/vod/test
Message	FileField was trying to display the file sites/default/files/137/1/20111119_176_test.mov, but it does not exist.
Severity	warning
Hostname	76.228.65.77
Operations

And here is the link via FireBug:

https://BUCKETNAME.s3.amazonaws.com/sites/default/files/137/1/converted/...

So Close!

Thank you for all of your help!

----Update----

It seems to be Flowplayer... And the built in HTML5 player?

First I switched to QT as the player (as I am ordering up MP4 files) and it worked!

So then I switched to SWFTools and pointed everything to hit JW (which I thought I had done before, guest not) and it worked!

Then I switched everything BACK to Flowplayer, and I get the white box.

JW is my player of choice, so I am not sure how where or why it was not set to be the default player to begin with.

I will continue to test, but hell yeah- thank you!

Next- What do you feel the issues are that would prevent the 4.6 Beta from being used in production? Now I am excited...

If it works for you and you feel confident about it you can use it in production if you want. I just added the warning message so users realize that this is a big update with lots of changes. Most 4.x -> 4.x+1 updates are small and changes only affect certain parts of the code. In this release I changed almost anything related to S3 and Zencoder. It could also have been a 5.0 release considering the amount of code that has changed, but as all changes relate to bug fixes and not to new features, a new major release is not appropriate, I think.

At this moment I don't know of any issues with 4.6, otherwise I would have already fixed them. I just want to let others test their scenarios for a week or so to catch bugs before releasing a final 4.6 version.

Excellent, thank you again, you rock.

Hi Jorrit,

When I delete a content with a video field (CCK/Video), I get this message: string(4) "4034"
It does delete the content but never returns to the main page and display a blank page. I thought you put some sort of "print" message for debugging. I'm using video-6.x-4.6-beta1 and this problem doesn't occur on video-6.x-4.6.

One more problem, when I enable "Bypass Video Conversion" from Site Configuration -> Video -> General, it still sends the job to Zencoder. I think when we select that option, it means we don't want to encode it via Zencoder or ffmpeg. Right? But when I check mark "Bypass Video Conversion" on the form that I'm creating content, it works fine and doesn't send the job to Zencoder. Although this option is working for me, it's much more convenient to set it permanently rather than on every content. I saw this problem on all versions I have been using lately. (video-6.x-4.6-beta1, video-6.x-4.6, video-6.x-4.5, etc)

Thanks Jorrit,

The debug line is my fault, thanks for telling me. I have deleted it.

About the bug: I'll try to fix this, but I also think that it was already broken before. Does the user that you use to upload videos have the 'bypass conversion video' permission?

>> About the bug: I'll try to fix this, but I also think that it was already broken before. Does the user that you use to upload videos have the 'bypass conversion video' permission?

I'm uploading with super user (admin). So, permission should not be an issue. Yes, this is not new bug with video-6.x-4.6-beta1. I've seen it in previous version as well.

Ditto on the "string" message.

A seperate but related question: I will check my config, but when I delete a node, it is not actually removing A) the pre-converted original and B) the converted file from S3.

I have noticed this in previous versions as well, where upon conversion, the original file is maintained in addition to the converted. Is this by design or a S3 permissions config issue on my end?

Please let me know if this is PEBKAC or if I should open a new issue.

Thanks!

jhrizz: What converter are you using?

The Bypass conversion setting on the Video administration screen should control the default value of the Bypass conversion setting on the video edit screen, however due to a bug in Drupal this did not happen (has something to do with the fact that the checkbox is added to the form dynamically). I have fixed this in 6.x-4.x-dev. Could you test it?

Zencoder.

I have not had time to get in front of the computer this morning to check Zencoders permissions (I am using Bucket Policies) but I know Zencoder has GET, PUT and ACL.

I know my S3 user has "full permissions" but I need to double check the policy.

Will get back ASAP.

Again, thanks for the help!

- jr

Hi Jorrit,

No, I don't see the default value of Bypass Conversion setting when I upload the video. Are you sure you uploaded your fix to 6.x-4.x-dev? The date of 6.x-4.x-dev is 2011-Nov-18 which is two days ago.

Jhrizz: I'm only talking about the Drupal permissions.

Naderat: I don't upload releases, the Drupal site creates them once a day when there have been changes. You need to wait for the dev release to be updated.

Hi Jorrit,

It works like a charm. Thanks for your great support. Is there any way we can compensate part of your effort through Paypal?

Ditto! Thank you for all of your help!

I have not downloaded the most recent DEV, however with Beta1, when a video node is deleted, only the original upload is removed from S3- the "converted" folder and the clip in it remain. This is after I ensured that Zencoder has Get, Put, Put ACL and Delete capabilities. Drupal's permissions on S3 (I believe) are handled via the native S3 credentials?

Know that I am using transliteration, so the file name underwent a change from the original.

Also, is there a way I may change the name of the folder that gets created to something other than "converted"?

- Cheers!

I think the transliteration module could very well be the problem. I'll try to reproduce it, but you may be able to test this by disabling it for a moment. Are the thumbnails deleted from S3?

The converted name will be there for a while. First the bugfixes, new features after that. What's the problem you have with this name?

naderat: I don't do this for the money, so maybe this is an alternative for you.

jhrizz: what was the original filename before you uploaded it?

I think the transliteration module could very well be the problem. I'll try to reproduce it, but you may be able to test this by disabling it for a moment. Are the thumbnails deleted from S3?

The converted name will be there for a while. First the bugfixes, new features after that. What's the problem you have with this name?

naderat: I don't do this for the money, so maybe this is an alternative for you.

Jorrit-

First off, consider the primary concern of this thread as "fixed" for me. All I have left is to test the version without the debug message and I am a happy camper.

Second, I post the concern regarding the improper deletion from S3 in the proper issue (I think I saw some replays come through recently).

Again, thank you, and you may indeed see some of those titles appear. May I also recommend for your wishlist: "Adventures of Apple Founder - Atari, Apple, Areospace and beyond" and "insolence of Office - Socio Politics, Socio Economics and the American Republic"

I have tested this myself now and my conclusion is that SWFTools and/or Flowplayer have difficulties with ampersands in URLs. If you can get Flowplayer to work using the Flowplayer API module, that would be the best solution in my opinion. I could try to supply a patch to Flowplayer, but the fact that they have a beta version outstanding since March and 140 open bugs suggests that the module is (almost) abandoned.

What video player in Drupal do you folks recommend to play MPEG4 (H.264/MPEG-4 AVC) and FLV as the fallback?

A fallback is only used for HTML5, all other players use just one file. I personally use the HTML 5 player with MP4 and Flowplayer with MP4 as fallback, because Flash can also play MP4 since version 9 or 10. The only issue is that Firefox does not fall back to Flowplayer when it can't find a suitable file format. You can fix that by using a small piece of Javascript that replaces the video tag by the fallback, or enable the Ogg preset. If you want to support very old Flash versions you still need flv.

I have tested this myself now and my conclusion is that SWFTools and/or Flowplayer have difficulties with ampersands in URLs. If you can get Flowplayer to work using the Flowplayer API module, that would be the best solution in my opinion. I could try to supply a patch to Flowplayer, but the fact that they have a beta version outstanding since March and 140 open bugs suggests that the module is (almost) abandoned.

A fallback is only used for HTML5, all other players use just one file. I personally use the HTML 5 player with MP4 and Flowplayer with MP4 as fallback, because Flash can also play MP4 since version 9 or 10. The only issue is that Firefox does not fall back to Flowplayer when it can't find a suitable file format. You can fix that by using a small piece of Javascript that replaces the video tag by the fallback, or enable the Ogg preset. If you want to support very old Flash versions you still need flv.

I can't get it to work with flowplayer. Keep spitting "stream not found but the URL in the object code looks fine. Can you let me know how you got that working in your site?

Do you have a public example of this problem?

I am using JW Player 5.x and SWFTools. If I switch from JW to Flowplayer, I do indeed get an error. Be aware that, depending on your cache settings, changes may not seem to take effect. Flush, reload, and it should kick in.

Flowplayer via SWF tools does not work in combination with private URLs. I intend to add a warning message for this. Flowplayer using the Flowplayer API module works fine.

naderat: I received a package from Amazon yesterday. Thanks a lot for the gift and the nice words that came with it.

The warning has been added.

if u are using flowplayers theme function in a customfilter you can get everything working in a very nice manner:

Features:
- upload via cck
- multiple [video:n]- Tags in ckeditor-bodyfield
- works with s3 private also

see this:

// node-object für pfad holen
$object = menu_get_object($type = 'node', $position = 1, $path = NULL);
$field = $object->field_lesson;

// ersten filepfad des nodes in variable schreiben 
$url1 = $object->field_lesson_s3[0]['filepath'];
//dpm($object->field_lesson_s3[0]);

// field-ID holen
$fid = $object->field_lesson_s3[0]['fid'];
$fid = intval($fid);

// abfrage ob fid in video_s3 tabelle
$result = db_query("SELECT * FROM {video_s3} WHERE fid='%d'", $fid);
$result = db_fetch_object($result);

// status für fid (10 wenn bereits übertragen)
$status = $result->status;
$status = intval($status);

if ((isset($result)) && ($status == 10)) {

$objectUri = $url1;
module_load_include('inc', 'video_s3', '/includes/amazon_s3');
$s3 = new video_amazon_s3;
$s3->connect();
$url = $s3->get_authenticated_url($objectUri);
$url = urlencode($url);
print_r($url);

  $video = theme('flowplayer', array(
    'clip' => array(
      'url' => $url,
      'autoPlay' => FALSE,
      'autoBuffering' => TRUE,
    ),
  ));

$result = $video;
}
else {
$base = base_path();
$url2 = $base.$url1;

  $video = theme('flowplayer', array(
    'clip' => array(
      'url' => $url2,
      'autoPlay' => FALSE,
      'autoBuffering' => TRUE,
    ),
  ));

$result = $video;
}