Integration with CDN Integration module

At this point I'm not rewriting anything - trying to figure out the endgame.

I did try the CDN module but ran into a different set of issues with it. The core module only currently supports 'origin pull' CDNs out of the box. What this means is that you rewrite all of your URLS from http://mysite.com/myfile.jpg to http://mycdnprovider.com/myfile.jpg. When the CDN gets that request, it acts like a giant proxy server in the sky. If it does not have the myfile.jpg cached it goes and gets it from your site, caches it, and then continues to serve it up. You basically don't have to do anything on your end to make that happen.

Amazon's S3 and CloudFront offerings don't work that way. They are 'push' CDNs, whereby you have to have your static content uploaded to their site before they will serve it. This means finding a way to replicate all of your static content up there in realtime or else you will get 404s. The CDN module has a system daemon to try to do this, but I'm not convinced it will work for my purposes. I have tons of Imagecache stuff happening on the backend and am looking to scale the site up pretty high so I don't see a good way to do this with fast enough response time with a push CDN. I don't want an imagecache to be created, user hits reload, and it doesn't make it to S3/Cloudfront in enough time to serve the file.

So where I am right now is looking for a way around that problem. My interim solution was to manually replicate static content to S3 that does not change often - CSS files, background images, etc. That's the low hanging fruit to achieve some of the benefit of CDN without going all the way and solving my push issue. To do that, I wanted to try rewriting of the URLs appropriately, but relative paths make that a big challenge.

My options boil down to -
1) Use the CDN integration 'module' / hack of core, but WITH the daemon and try to make it work and hope it replicates quickly enough to push content to Amazon

2) Find another CDN provider that uses origin pull

3) Hack up some type of rewriting option, which is where I am leaning now.

Thanks for the pointer to that function - it may be very helpful. I also found this link which talks about a rewrite at the theme layer for this: http://www.opensourcery.com/blog/dylan-tack/content-delivery-masses

Interesting - there is also an Apache module mod_cdn which could help with this. It seems to perform some of the rewriting magic for static content. If the directives can be used with in httpd.conf that may help rewrite the URLs assuming it gets the http/https part correct.

http://www.voxel.net/labs/mod_cdn

Another thought. Sorry if I keep populating your issue queue with my random musings, but perhaps this information will help someone else at some point in the future.

This is an interesting option:
http://code.google.com/p/s3fs/wiki/FuseOverAmazon

It's a FUSE module which allows for the mounting of an Amazon S3 bucket as a local filesystem. The idea is to create the bucket at Amazon and mount it in a specific location locally (perhaps as part of the location of the css/images for theme files or as the /path/to/your/files for imagecache and other items). Then Drupal should be happy when static content changes or is created since it is just doing normal filesystem writes, but they go to S3 automagically with no python daemons (from CDN module) or trickery. As long as the URLs are rewritten properly the static content would be served from the CDN.

Anyone who tries this who is not deeply technical should make very sure to not replicate .php or other non-static files to the CDN. If the mimetypes on the CDN change it could allow people to download your source code. I'm only thinking of doing this for .ico, .css, .png, .jpg, etc.

This is now the option I am pursuing - the Drupal CDN module + the S3 FuseFS with a specific path for static content and using S3 (both HTTP and HTTPS) to distribute the files. When CloudFront supports SSL I'll use that to serve up the files.

More notes -

There are multiple projects all called "s3fs" but the C++ project referenced above seems to work the best for me so far. To get it going, I compiled the code, created a file called /etc/passwd-s3fs with my S3 public and secret key (separated by a colon), and then used this command line to mount the filesystem:

/usr/local/bin/s3fs mybucketname -o default_acl=public-read -o retries=15 -o connect_timeout=8 -o readwrite_timeout=40 /mnt/s3

... and now /mnt/s3 is mounted locally. I did not enable caching because I am just writing to the S3 bucket and not reading from it.

Note the 'public-read' ACL option. This means ALL of your content is readable by anyone on the Internet at the URL http://s3.amazonaws.com/yourbucketname/yourfilename and https://s3.amazonaws.com/yourbucketname/yourfilename. Again, this is why I am using S3 to host static content since it works well with SecurePages by allowing transparent switch from http to https.

Now that the S3 bucket is mounted as a local filesystem, I can move data to it through lots of different mechanisms. I am now evaluating the best way to do live synchronization to the S3 bucket and so far the leading contender is lsyncd: http://code.google.com/p/lsyncd/

Stay tuned. This might just work out well.

For those that have read the above thread, here are things to know.

Ignore everything I suggested about s3fs and lsyncd. Don't waste your time - instead definitely visit the Drupal CDN module page and use the 'daemon' program written by that module's author. It will not only upload your files to the CDN, it will also integrate the CDN properly with Drupal so that if a file is not yet on the CDN, it is served locally. That way you never get broken images or links. It's excellent.

Now - on to integration with SecurePages. I have created an issue in the CDN module queue here: #568954: HTTPS support. This issue also includes a new patch for the CDN integration module which disables CDN serving of static content on pages which are served as SSL by SecurePages. Why? Because otherwise your SecurePages SSL page would have non-SSL content from the CDN and either give you the 'broken lock' in Firefox or the big nasty "some content is insecure" message in IE. No good.

So the idea with that patch is all non-SSL URLs have static content served from CDN, and SSL URLs have static content served locally from the Drupal server.

Marking this as a dupe since I created the other issue. See #568954: HTTPS support for details.

Don't bother with s3fs or anything else except File Conveyor. Trust me - I wasted a ton of time. Otherwise you will get the content synced but have no way to tell Drupal about it. File Conveyor hooks in to the CDN module in the correct way and allows the site to dynamically serve content out of the CDN only when it was successfully uploaded.

File Conveyor is the daemon specifically written to integrate with the Drupal CDN module. The reason you want to use it is because it tells Drupal what files have been synced to the CDN and what files have not. It can push directly to S3 and Cloudfront and leverages operating system kernel calls to monitor the filesystem. You won't need s3fs or anything else to move data.

If you use s3fs, Drupal has no idea what is local and what is on the CDN. Let's say a user uploads a photo. Drupal wants to then take you to the node right after the upload to show you the created file and resulting node that was created. Since we are dealing with a push CDN there is a delay between the file arriving on your Drupal box and being pushed to S3. With s3fs, you run a significant risk of the browser serving an image-not-found because it hasn't yet been moved to the CDN. The CDN module won't know it hasn't arrived yet, and it will serve up a URL like http://s3.amazonaws.com/mybucket/myfile.jpg which does not work because the file isn't there. It's a race condition.

File Conveyor solves that problem by managing the upload process to the CDN but also maintaining a small SQL database of which files were uploaded and which ones were not. Before Drupal provides the user with the URL to the file, it checks the database to see if it is on the CDN. If not, it serves up a local URL. So none of your pages ever will return with a file-not-found error.

File Conveyor can also preprocess every file - i.e. you can run yuicompressor on javascript, strip out whitespace from CSS files, and even rename each graphic image to allow for setting very far future expires headers without having to deal with the file renaming game.

I really recommend looking at this if you are going to be using the CDN module - especially with a push CDN.

http://github.com/wimleers/fileconveyor