Sharing files across mutliple EC2's

slamdunk - May 16, 2009 - 17:29

Hello,

Wanted some suggestions as we need to share a large set from /files amongst many front end EC2's. We were using NFS, but too many problems with locks. We have content providers, and therefore upload content (or update) and these assets (<800kB) get stored within /files.

I noticed a new project called Bitcache (http://drupal.org/node/336715) that will support S3. Most of these files on S3 will need to authentication fist before any user can download them. As we expect the /files repository to grow drastically, should we be considering EBS on EC2, and some method to fetch the files when requested by maybe serving to S3? MemcacheD don't think will work with such a large repository. MogileFS is similar to S3 from my understanding and apparently not so easy to setup.

Also noticed project http://drupal.org/project/cdn but seems still under development. Another one is http://drupal.org/project/media_mover under consideration.

Noticed this too, http://www.persistentfs.com - its only good for read only. If somehow we could get from major repository (EBS?) and copy to persistentFS upon request so to be shared amongst Drupal Front ends?

Btw: using Rightscale.

So, any ideas how we could move away from NFS locks or any suggestion will greatly be appreciated how we can narrow down our options.

Thanks
G

I can't respond fully

jtbayly - June 23, 2009 - 19:07

I tried project/cdn, and it seemed quite promising, but I ran into a bug in advanced mode (required for S3), and since it isn't actually on D.O right now, it seemed that I had to move on for now. For the record I think this looks like the most promising solution in the long run.

In the meantime we are using media_mover, which has distinct advantages for us (because of its ability to process files before uploading them). I'm happy with it so far, but we aren't done yet. The next version looks much more flexible.

We were specifically looking at those modules that support Amazon S3 because that was a requirement for our project.

fwiw,
-Joseph

cdn is now on d.o.

mooreds - August 14, 2009 - 23:20

First release was July 1, 2009. Haven't tried it yet, but here it is: http://drupal.org/project/cdn

 
 

Drupal is a registered trademark of Dries Buytaert.