Thought I'd mention that CloudFront now supports custom origins, so in theory you can use this module now without file conveyor. The announcements were on their web page, but this was the best documentation I could find:
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=5429...

I used the cfcurl.pl script from here:
http://aws.amazon.com/code/developertools/1878

with the key file called $HOME/.aws-secrets, formatted like this since they give you zero documentation in the above link (scroll about 1/2 way down, to step 2):
http://www.thebitsource.com/infrastructure-operations/amazon-cloudfront-...

and my xml distribution file looks like this:

<DistributionConfig xmlns="http://cloudfront.amazonaws.com/doc/2010-11-01/">
  <CustomOrigin>
    <DNSName>myrealwebsiteurl.com</DNSName>
		<HTTPPort>80</HTTPPort>
		<OriginProtocolPolicy>http-only</OriginProtocolPolicy>
  </CustomOrigin> 
  <CallerReference>0123012310230123</CallerReference>
  <CNAME>mycnamealias.myrealwebsiteurl.com</CNAME>
  <Comment/>
  <Enabled>true</Enabled>
  <DefaultRootObject>index.php</DefaultRootObject>
</DistributionConfig>

and an upload command line of:

cfcurl.pl --keyname primary -- -X POST -H "Content-Type: text/xml; charset=UTF-8" --upload-file thexmlfilefromabove.xml https://cloudfront.amazonaws.com/2010-11-01/distribution

That gets it uploaded to Cloudfront properly. You can then see and mange the distribution in the AWS management console. All of the above nonsense is required because the console does not yet support this via web config.

I'm still working on testing it. I'll report back with more as I go. I'm not asking for help here, just posting an issue that may help others later.

Comments

rjbrown99’s picture

Status: Active » Fixed

That was pretty much it - seems to work without a hitch. Nice!

wim leers’s picture

Title: Cloudfront now supports custom origins (i.e. origin pull!) » Amazon CloudFront distribution creation support (CF now supports Origin Pull)
Version: 6.x-2.0-rc5 » 6.x-2.x-dev
Component: Documentation » Module
Assigned: Unassigned » wim leers
Category: support » feature
Priority: Normal » Major
Status: Fixed » Postponed

Hehe, I'm active in that same thread you're referring to :) ;)

Also, that Perl script didn't work for me, I guess I broke my Perl installation. I wrote a script that uses the AWS PHP SDK (https://github.com/amazonwebservices/aws-sdk-for-php) instead. (Use version 1.1.0 or newer, 1.1.0 was released yesterday and includes support for the new Custom Origin functionality.

I'll add the ability to create this through the CDN module itself in the near future, in a post-2.0 version.

porcupine73’s picture

Interesting news about CloudFront offering a pull mode. I'd be interested in having this support in CDN module when my sites start getting more traffic. I used CloudFront on a project last year (but it was vBulletin not Drupal) and was very satisfied with the performance.

rjbrown99’s picture

The support is there now - it 'just works' once you set up the custom origin in cloudfront. I'd be interested to see what kind of performance you could get out of am Amazon micro instance using a cloudfront custom origin. I'm betting it would be decent for a very low end hosting site.

wim leers’s picture

Performance is looking very promising… I've also switched http://driverpacks.net to use CloudFront instead of MaxCDN. That site has visitors from all over the world (from virtually every country there is), so it greatly benefits from the larger number of edge locations (PoPs) when comparing CloudFront to MaxCDN: lower overall latency is ensured.

Also, the latency for my location (Hasselt, Belgium) is much lower with CloudFront than with MaxCDN (although both have a PoP in Amsterdam).

vacilando’s picture

Excellent! Subscribing.

stevenator’s picture

This is fantastic...Subscribing~

stevenator’s picture

@rjbrown99 just wanted to say thanks again for posting this info. I have completed the steps using the cfcurl.pl method and all works with a working perl installation :).

rjbrown99’s picture

No problem, glad it worked for you. It took me a good hour of poking around to find the right way to do it since it isn't yet in the AWS Management Console. I also agree with Wim - so far it has been performing quite well.

wim leers’s picture

I was looking into integrating this with the CDN module just now, and I noticed that the AWS PHP SDK is licensed under the Apache 2.0 license. Which is compatible with GPL v3 (*not* v2!), but Drupal CVS requires GPLv2. Which would imply that I'd have to require the end user to download and install the AWS PHP SDK separately. Kind of stupid.

rjbrown99’s picture

How about asking on the AWS discussion forum if they would dual-license it? Amazon is generally pretty responsive, and having a major project like Drupal better integrate with their platform means they have more of an opportunity to make money. You may get a positive response.

wim leers’s picture

It's worth a shot I guess :)

wim leers’s picture

Terko’s picture

Please, tell me how to set expire date? All my images with custom origin are without this setting and the users download them again and again

wim leers’s picture

Status: Postponed » Active

@teri: Please don't hijack issues. That's *not* nice. You don't have to make any special CloudFront settings, you just have to make sure your origin server (i.e. your Drupal site) sends out the right headers. I.e. Cache-Control or Expires headers.

Terko’s picture

Excuse me and thank you for the reply.

chrism2671’s picture

Hi Wim,

What about doing the 'usual' license incompatibility solution (end user has to download and install files themselves), a la http://drupal.org/project/jquery_ui ?

I think that would be a really acceptable inconvenience, considering all the benefits to be had from easy set up!

chrism2671’s picture

Alas, I tried this perl script and floundered too :-(.

chrism2671’s picture

Let me add, Cyberduck for Mac has a GUI for configuring Cloudfront custom origin CDN. Not tried it yet but it looks promising.

chrism2671’s picture

OK Just tried this, it DOES work but it is a bit hairy. You need to update to the nightly build to make this work. Does not support any port other than 80.

wim leers’s picture

@chrism2671: letting the user download the lib is the only alternative indeed.

naeluh’s picture

@chrism2671

Can you provide a little insight as to how you got around the hairyness of the cyberduck gui.
I have downloaded the nightly build, however I am having no luck accessing the proper dropdown menu?
I have tried to change the web url per the documentation on cyberduck but it is not allowing me connect on any ports?
I have tried port 80. Is there a way to access the change of the web url that I am missing?

thanks

rjbrown99’s picture

The command line really isn't that bad if you guys want to venture there to give it a try. The hardest part was figuring out the perl script, authentication, and XML stuff. My initial post should get you up and going and it might be easier than slogging through new GUIs.

chrism2671’s picture

@naeluh Yes. Create a connection to your WEBSERVER where your html and so on would normally be, and click File->Info. You'll see the settings in there.

@rjbrown99 Did you do it on a mac or under linux?

rjbrown99’s picture

I did it under Linux, but it shouldn't matter either way. Both have perl, and if you are missing modules you can just use CPAN and install them.

naeluh’s picture

@chrism2671

Thanks, I think I miss spoke:

I can access the info area, but when I choose the drop down menu DELIVERY OPTIONS it shows only 2 options:

Download (HTTP)
Streaming (RTMP)

The option for Custom Origin Server (HTTP/HTTPS) is not available?

thanks

naeluh’s picture

@chrism2671

I see now . Thanks. you dont connect thru the s3 server, you select the your own server I get it.
thanks again

ckreutz’s picture

@rjbrown99 thanks a lot. The solution with cfcurl.pl worked like charm. I created the distribution using a mac. Maybe worth to mention that cfcurl.pl generates a new distribution in cloudfront. That can be deleted any time. I did not understand first that it will be visible in the console.

Does such a custom origin cloudfront approach can compete in terms of speed with an Amazon S3 bucket? It still needs to be pulled from the original server and that is often not as fast as an Amazon bucket? Thanks in advance for any hint.

wim leers’s picture

ckreutz: CloudFront should be much faster than S3, but it of course depends on your location. Once every 24 hours, CloudFront will fetch the file again from the origin server you've defined. The initial access might take relatively long, because CloudFront needs to go to the origin and fetch the file. How long that takes, depends on the speed of the origin server and on the size of the file, of course.

ckreutz’s picture

Thanks Wim for your explanation. Guess now I understand it better. ;-) And thanks for your great CDN module.

zach harkey’s picture

Cyberduck has to be the easiest way to set up a CloudFront distribution using a custom origin pointing to a drupal site hosted on your own server.

Obviously you have to have an AWS account, and you need to be signed up for CloudFront, but you don't use the AWS Console to create the distribution since the console will force you to choose an S3 bucket. Instead, use Cyberduck (Be sure to install the latest beta release) to log in to your drupal web server using SFTP (SSH File Transfer Protocol). Once connected, navigate to your document root folder and select it. With your document root folder selected, select File > Get Info.

A new pop window will appear where you enter your AWS account credentials. Once logged in, check the box to "Enable Amazon CloudFront Distribution". This will deploy the CloudFront distribution using your document root as its 'bucket'. Keep in mind that this part can take about 10 or 15 minutes during which the Origin and Where URLs may be wrong. Don't keep clicking around or you can easily launch multiple distributions, just leave the panel alone and head over to the AWS control panel where you can check on the status of the distribution and and specify a CNAME if you want.

porcupine73’s picture

I switched my sites over to using Cloudfront via a custom origin to my Web server. It works very well and was pretty easy to set up. Then I pointed the CDN module to my cloudfront URL's (which I CNAME'd to my own subdomains). I used BucketExplorer to set up the custom origin, which was a little tricky in version 4.0.3 but it does work just fine. The Cloudfront does seem to be very fast. You do need to make sure your server (or whatever custom origin you are using) emits the desired EXPIRES headers; otherwise I'm not sure how often Cloudfront will come back to the custom origin to check for a new version.

Also I did some tweaking on my custom origin so that the requests are not passed through Drupal, because for some reason a couple times Drupal returned the 404 custom search page for an image that couldn't be found, and that kind of messed things up. So if the file is not found I return 'temporarily not available' (I forget the code, 405 or 406 I think it was) to Cloudfront now instead. Also I had some issues with images from imagecache, because for the image to be generated, the request must first come through the actual source URL, and I was going around that. (I have about 12 sites set up as multi-site, but I use the same Cloudfront distribution to serve them all).

wim leers’s picture

There's now an AWS SDK module: http://drupal.org/project/awssdk, but it requires the Library API *and* Drush Make. That's a whole lot of dependencies.

BManuel’s picture

Thanks
A friend just told me to use drupal(my domain) as "custom server" for the bucket and Amazon will
automatically pull it in. I just did that, Now I am hoping he is right.
I also followed the sample config.xml as backup.

"You need to set up origin pull ("CloudFront Custom Origin") at AWS side first -- read their manual. Then they know where get the contents from.
You can use the free http://cloudberrylab.com/default.aspx?page=cloudberry-explorer-amazon-s3 to set it up." message from Tomáš Fülöpp (vacilando.org)
Thanks Robert J. Brown for your help too

wim leers’s picture

Well … yes, that's exactly what this issue is about…

I'm not sure what information you're trying to add here.

cartagena’s picture

Hi--most of the above discussion is above my head...but I have set up my S3 and CloudFront to host videos. I set up S3Fox, all is good. Marked my video public and got the distribution URL. Using media module on D7 I put it on my site. I'm using mediafront and OSM player. Video plays but there is no audio. I would be extremely grateful for any suggestions. Thank you.

capella1234’s picture

Folks - All this gimmick is not necessary any more. Now you can easily create distribution with custom source all at web gui. Check the doc here - http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide...

It took 15 minutes for me to get this working. I have written a blog post also about it - http://www.dineshmaheshwari.com/2011/04/joomla-cdn-amazon-cloudfront-wit...

wim leers’s picture

Hurray! Thanks for reporting, capella1234 :)

cartagena’s picture

Thanks, very helpful!

wim leers’s picture

Title: Amazon CloudFront distribution creation support (CF now supports Origin Pull) » Link to Amazon CloudFront custom origin docs
Component: Module » Documentation
Category: feature » task
Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

hansrossel’s picture