As mentioned in another issue, I am rewriting the Amazon S3 module for Media Mover. Some of the problems with the current version are:

  • Dependant on PEAR libraries
  • Does not stream files
  • Does not use Expect 100 headers
  • among others.

Going into this I realized that Drupal did not yet have a Amazon Web Services module (there are like four just for ecommerce service but no others) and I have seen several modules trying to write their own implementations to use these services. So I have begun writing a module that abstracts the basic calls to Amazon for these services. AWS Module

Since media mover deals with large files, streaming was a requirement. After looking at several of possible REST implimentations, none of them were capable of streaming or capable of handling Expect 100 headers. This includes the natural decision of drupal_http_request(). Thus the REST Client module was born.

I have laid out this story to show my current work and the path I am on. I have created this issue task to collaborate about the rewrite of the mm_s3 module using these base modules.

Please check out the other modules and post any thoughts or requests you may have about the mm_s3 rewrite here.

P.S. Constructive criticism welcome.

Comments

arthurf’s picture

arthurf commented

All thumbs up from me. I really like seeing the AWS module. Thanks for pushing this along! Now all we need to do is make an EC instance for FFmpeg, and this will be a rather nice solution :)

dragonwize’s picture

dragonwize commented

That is in the works. I plan on writing an mm_ec2 module and creating a public AMI as soon as I have the S3 part done. Then I will be writing a SQS functionality to tie it together and enhance it. All of this I need for an up coming site.

But help is always welcome :) if anyone wants to collaborate.

budda’s picture

budda
Primary language English
Location Cheshire
commented

@dragonwize / arthurf - we've just had a meeting with Jeff Barr at Amazon as he was in the building this morning.
We mentioned our use of MM/S3, as well as discussed some of the feastures needed for S3 in the future.

He suggested Amazon sponsoring the update proposed by dragonwize, either finacially, or through delivery of pizzas ;-)
If you'd be interested in this drop me an email via the drupal.org contact form and I can hook you up.

Would be good to get an updated streaming version of the MM S3 component. For now we've just increased the php memory limit to upload 64MB files. Which isn't a great long term solution.

arthurf’s picture

arthurf commented

I'd LOVE to see that happen. Also - EC2 integration.... If amazon would be willing to do this, I could try to help dig up some resources, or maybe even make it an SOC project?

dragonwize’s picture

dragonwize commented

I agree, that would be great.

I am currently in the middle of a rewrite of my REST Client module because my first attempt proved to not to be very efficient and had too many bugs. I am writing it in OO style now like I wanted to before but I was trying to keep it in line with Drupal coding. It still has a Drupal like function interface so you don't interact with the OO part directly. And I am still making it PHP4 compat (until the D7 version).

The AWS S3 module is 80% done. I have a couple functions left and some helpers left to write as well any changes required by the REST Client rewrite. The EC2 and SQS modules also have had their first drafts written by simonc.

So once that is done and some testing proves that is a stable implication for all the HTTP features. I will rewrite the mm_s3 module, which shouldn't take a whole lot at that point. Then on to writing mm_ec2 and mm_sqs modules.

I have also been collaborating with the other AWS ECS developers, including Jeff Eaton. So far I have got most of them to switch their modules to be listed under "Amazon Web Services" grouping for the Drupal module page and we are discussing any areas of possible duplication. Jeff has also be kind enough to offer his help in reviewing our code and offering advise.

Any help Amazon could offer would be greatly appreciated whether it is financial or not.