S3 enhancements - move original, set ACL, etc.
| Project: | FlashVideo |
| Version: | 5.x-2.x-dev |
| Component: | Code |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | needs review |
These patches (against Revision 1.20.2.39 in CVS) are a first draft attempt at extending the functionality of the S3 plugin. Would appreciate any feedback. I had to fiddle with flashvideo.module as well as flashvideo_s3.module to get this working.
This adds:
Move original files (along with converted files and thumbnails) to s3. Currently this is an all or nothing option (if s3 enabled, all are moved).
If 'Delete Local Files After Move' option is checked, original file is deleted also. Currently this will not play well with the 'Delete Original Video' option in FFMPEG settings, so leave this unchecked for now.
Additional checkbox with the option to make original files ACL Private. Kind of useless at the moment unless you just want to back up original files somewhere without public access. You need to set Original Directory in FFMPEG settings to something different than Output Directory as the S3 bucket name is based on those settings.
To Do -
Files didn't seem to be getting deleted from S3 on node deletion when 'Delete Local Files After Move' was checked. Not sure if this is something I introduced or it was inherent in the old module. Need to test further.
Hook into "Download Original" link generation to allow for generating an expiring link when original file is set to private ACL.
Possibly enable CloudFront CDN checkbox (will probably need to use to different s3 drivers).
Need to test with extremely large videos. All my tests have been 5mb or less.
Better way to do flashvideo_s3_check_original() in flashvideo_s3.module to accommodate mp4 streaming mods, etc....
Thanks Travis for a really incredible (and extensible) module!!!
| Attachment | Size |
|---|---|
| flashvideo_s3.module.patch | 4.11 KB |
| flashvideo.module.patch | 2.7 KB |
#1
(fixing status)
#2
Great enhancement. Although, this will probably take some major testing before it makes it into the release. These days I have to be very careful when making changes since so many people are affected. Probably will not make it for 1.4, but maybe it will make it for a beta release for 1.5.
Thanks for your efforts.
#3
Yes, definitely needs a lot of work/testing. I wouldn't recommend anyone try this on a production site.
These enhancements are something we need for a site launching in December, so I just thought I'd share the patches in case anyone else had similar ambitions (and hopefully to get some help with testing!).
Was hoping not to touch flashvideo.module, but ended up having to add a hook call and move something to another function to prevent using up too much memory (original file rename from flashvideo-convert to _flashvideo_perform_postop).
I'm hoping to post code today that uses S3's Query String Request Authentication to generate an expiring download link if the user has set that bucket to private.
The QSRA generation function is working, but I'm having trouble tying into the download original link generation without modifying flashvideo.module... if you have any suggestions I'd love to hear them.
#4
Files didn't seem to be getting deleted from S3 on node deletion when 'Delete Local Files After Move' was checked. Not sure if this is something I introduced or it was inherent in the old module. Need to test further.
Yeah, um. Don't use this yet. The changes in flashvideo.module are preventing the .flv and .jpg from being attached to the node correctly even when s3 is turned off. Will post a fix as soon as I figure out why.
#5
After some investigation (including eventually just re-imaging my entire server from a backup), it seems this was a bug in the module before I started tinkering. This (http://drupal.org/node/336299) issue seems to impact 5.x-2.8 as well. Will submit it as a bug under a separate issue.
Files moved to s3 that are not listed as an attachment will not be deleted upon deletion of the node until this is bug fixed. They may also leave some database remnants (flashvideo, flashvideo_s3, and {files}) as well that you would have to clean up manually.
If anyone is testing this, I highly recommend the S3Fox plugin that will let you go in and delete the S3 files manually.
These patches add to the above a working function to generate the Query String Authenticated URL, modified slightly from http://undesigned.org.za/2007/10/22/amazon-s3-php-class.
You'd have to call this from your theme using something like:
l('link text', flashvideo_s3_getAuthenticatedURL($bucket, $uri, $lifetime))
$bucket is the name of the bucket (default is sitename + filepath, but you can use S3Fox to find it if you are having trouble)
$uri is the filename
$lifetime is when you'd like the link to expire in seconds from now
#6
This patch to flashvideo.module fixes the issue with file deletion from S3.
http://drupal.org/node/338211#comment-1141187
These updates now do everything we needed them to do... is anyone interested in me trying to extend this further? Was thinking about allowing for selection of what filetypes you'd like to move to S3 (images/original/converted) and being able to activate Cloudfront CDN for any or all of those filetypes.
We don't need those features right now though, so if there is no interest I'll leave it as is for the time being.