By rethinkwebdesign on

Ok, I'm super pumped right now. After a day of trial and error with a dozen different modules that try to integrate Amazon S3 as a way to store files and serve them through drupal, I THINK I FOUND THE DREAM SETUP!!!

Right now I have a completely seamless integration with my s3 bucket as a private file system for my Drupal Video Website. Files are upload straight to Amazon and then served straight from Amazon just like files on the local server.

Prerequisites:

- Dedicated or cloud (VPS) server. I am using 1and1.com Cloud Server $50/month and I host a bunch of sites on it. www.1and1.com?k_id=13279938

- An Amazon s3 account. Super cheap and awesome. Leave you settings on s3 to private so that your files are not accessible without authentication.

Then do this...

1) For private file system I use .htaccess. This one line that makes sites/default/files/private/* authenticate through drupal...

RewriteRule ^sites\/default\/files\/(private\/.*)$ index.php?q=system/files/$1 [L,QSA]

2) On my server I setup s3fs. Instructions here http://code.google.com/p/s3fs/wiki/FuseOverAmazon

Make sure that when you mount your s3 folder you use the "allow_other" switch and that your mount point is inside the default file system folder and inside the "private" folder we designated in .htaccess. I change the owner too. Obviously this is the structure for my server, please change the paths as needed...

mkdir -p /var/www/vhosts/mysite.com/httpdocs/sites/default/files/private/s3

chown apache:apache /var/www/vhosts/mysite.com/httpdocs/sites/default/files/private/s3

/usr/bin/s3fs mybucket -o use_cache=/tmp -o allow_other /var/www/vhosts/mysite.com/httpdocs/sites/default/files/private/s3

Make sure that the path for you CCK field is set to "private/s3" or something inside s3 like "private/s3/videos"

Done.

What do you think?

⋅ Categories: Drupal 4.5.x or older, Drupal 4.6.x, Drupal 4.7.x, Drupal 5.x, Drupal 6.x, Drupal 7.x

Comments

rethinkwebdesign’s picture

rethinkwebdesign commented

I can also just point my auto backup and migrate to my s3 folder!

lismail’s picture

lismail commented

I have no experience with s3, but I interested to know more.

Beside those settings above, do you think we also have to set the permission (chmod) of "files/private" or "files/private/s3", for example to 760, in order to block outbound access to these directory & content?

rethinkwebdesign’s picture

rethinkwebdesign commented

Not sure if that is necessary. Whatever the setting you would use for private file system if they where implemented.

dotsam’s picture

dotsam commented

I share your excitement for getting this working -- what a fabulous set-up!

Using your excellent instructions I have got this working on Ubuntu 10.04.1. For Ubuntu-specific installation of s3fs I followed the walkthrough here: http://www.logicwizards.net/2011/01/30/mounting-an-s3-bucket-on-ubuntu-l...

I am using the private_download module http://drupal.org/project/private_download to manage my .htaccess, and it works flawlessly so far.

Thank you for sharing this!

designguru’s picture

designguru commented

Okay - this is certainly the most straight-forward way to use S3 to replace the local files directory on a Drupal site I've seen out there, and I've been playing with it but I'm having trouble with imagecache - I set the site's File System Path to use sites/default/files/s3 and made 's3' my mounted bucket...

Files uploaded through filefields/imagefields seem to be loading okay but for some reason the imagecache module isn't creating a directory and sub-directories as needed in the new 's3' location - instead its created an empty file called 'imagecache'... so I'm thinking there's a problem creating sub-directories or something...

Any clues to get this working would be appreciated! :)

Thanks,

Qasim

Qasim Virjee
Principal, http://designguru.org
qasim@designguru.org
1.416.777-1864

wayn’s picture

wayn commented

I had done all by your article.
This is my site works:
1. I enable the s3fs on my ec2 server, and mount mybucket to files/private/s3.
2. I change my cck filefield path to /private/s3.
But when I upload a file to drupal.I found this file also in files/private/s3, and in my s3 bucket too. and the file download link also my drupal link,not the s3 link.
I just don't konw is the run right, I want it may upload file straight to amazon s3,and download from s3.

rethinkwebdesign’s picture

rethinkwebdesign commented

Check the setting for mounting the s3fs to make sure there aren't any other switches that may affect this.

jordanmagnuson’s picture

jordanmagnuson commented

Thanks for sharing this awesome setup! Using this with D7. I've got everything working, but there's one issue: Pecl upload progress does not work with my s3fs directory: the file uploads, but the progress bar jumps from 0 to 100% upon file upload completion, rather than showing the progress along the way. Obviously a bit of a problem for large file uploads.

Any ideas on how to get upload progress to work with s3fs?

PS: pecl upload progress is tested working correctly for everything but s3fs directories.

rethinkwebdesign’s picture

rethinkwebdesign commented

make sure you are using this trigger...

-o use_cache=/tmp

Other than that, make sure your pecl upload progress is working in other directories.

Hope that helps,

-Dan

jordanmagnuson’s picture

jordanmagnuson commented

Thanks for the help Dan, I really appreciate it. Sorry that I haven't replied before now: I got onto another project, and was expecting to get an email if a reply came here, but that didn't happen, so I didn't see your reply until now...

Anyway, I have tried adding "use_cache" to my trigger, but it doesn't seem to be helping with the progress bar issue. Here's what my fstab entry now looks like:

s3fs#pixel-scrapper /srv/www/pixelscrapper/public_html/sites/default/files/s3 fuse defaults,noatime,use_rrs=1,allow_other,uid=33,gid=33,use_cache=/tmp,default_acl=public-read 0 0

Do you have any other ideas? Have you seen progress bars working in your use cases? I would really love to use this setup, but the progress bar issue is a big one for me, since my site is based around user-generated content, and obviously people need progress bars when submitting large files...

I'll check back for your reply sooner this time than I did last time! :)