Wim - Thanks in advance for the module development. Looks great, and I am currently working to configure 'daemon' to work with Amazon S3. I have a request - the documentation, sample config file, and your thesis reference S3 support but there are no config examples.

1) Would you mind posting a config.xml sample for S3 - specifically the section and directives? I don't even need it explained - just the actual example is plenty for me to figure it out.

2) Do I need to configure boto outside of daemon and make sure it is working first? Or will the daemon just use config.xml directives and work properly?

Thanks in advance.

Comments

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Active » Fixed

1) If configured incorrectly (or incompletely), then the daemon should throw errors, telling you what settings are missing.

The <server> section should look like this:

    <server name="s3" transporter="s3">
      <bucket_name>yourBucketName</bucket_name>
      <bucket_prefix>youPrefix</bucket_prefix>
      <access_key_id>yourAccessKey</access_key_id>
      <secret_access_key>yourSecretAccessKey</secret_access_key>
    </server>

The remainder of the config.xml file remains identical, so you can look at the sample config file.

2) No, you don't need to configure boto outside of the daemon. Boto is included in the daemon and is instantiated and thus configured directly by the daemon, by using the settings from the config.xml file. So just using the config.xml file directives is sufficient!

rjbrown99’s picture

rjbrown99 commented

Thanks Wim. I have it working now, and I am on Fedora 11 with Python 2.6. I know the docs all say 2.5, but so far for me 2.6 is doing its thing with no problems. One note for anyone else who comes across this post - if you want to sync your CSS files to the CDN, make sure to specify a documentroot or basepath. Otherwise they are skipped during processing.

I may roll all of this into a handbook entry as I have a ton of thoughts about this. I also started down a very different path with mounting S3 and using lsyncd to replicate but there are significant drawbacks to using that approach vs. using your daemon. To save others a lot of time - if you want to integrate with a push CDN, start with Wim's daemon module first. The integration with Drupal is very key and you may not realize it at first.

One of the other things I may do is to modify the CDN integration code to support SecurePages. That module switches back and forth between SSL and non-SSL mode. There is no Amazon Cloudfront CDN for https, so what I want it to do is when serving non-SSL content use the Cloudfront CDN URL, but when serving https/SSL content use the S3 URL. S3 is not a CDN but it does provide offloaded image hosting and won't break the requirement to serve images/static content in the same delivery method as the page itself. I am thinking this requires a new form entry for CDN integration for the http vs https URL, and then an if statement somewhere in the code that calls this SecurePages function to tell if the page is secure or not: http://api.drupalecommerce.org/api/function/securepages_is_secure/6-4.

Something like "if securepages module exists and page is secure, use URL#1, otherwise use URL#2".

I also have a big thread going over in the SecurePages queue about this: #563896: Integration with CDN Integration module. I'll take a crack at solving it this weekend.

I posted a new thread about this as a new issue.

rjbrown99’s picture

rjbrown99 commented

Interesting. I am getting close - daemon is syncing files to S3 and seems to have built itself a nice little local database. I'm now having a different yet interesting problem. S3 is returning all of my files with a content type of octet-stream instead of image/jpeg or whatever the correct type should be. This results in the browser interpreting the images or javascript as a downloadable file rather than something to take action on.

There is a thread about it here: http://developer.amazonwebservices.com/connect/thread.jspa?messageID=59607

Perhaps the daemon and S3 transport are not setting the proper mimetype on the files during upload.

This isn't related so I will split it into a different issue.

rjbrown99’s picture

rjbrown99 commented
Title: Config.xml example for S3 transport » Config.xml example for S3 and Cloudfront transports

For the record, here is my config.xml snippit that is working for Amazon Cloudfront:

  <!-- Servers -->
  <servers>
    <server name="cloudfront" transporter="cf" maxConnections="5">
      <bucket_name>mybucketnamehere</bucket_name>
      <access_key_id>myaccesskeyidhere</access_key_id>
      <secret_access_key>mysecretaccesskeyhere</secret_access_key>
      <distro_domain_name>WEIRDBIGNAMETHEYGIVEYOU.cloudfront.net</distro_domain_name>
    </server>
  </servers>

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented
Category: support » task
Status: Closed (fixed) » Needs work

The example in #4 needs to be added into the daemon's README.txt, which does not mention <distro_domain_name>

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Needs work » Closed (won't fix)

I've moved the File Conveyor daemon to its own project at github: http://github.com/wimleers/fileconveyor. Please create a new issue there: http://github.com/wimleers/fileconveyor/issues.

Won't fixing it here.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Closed (won't fix) » Fixed

I committed this already anyway, to prevent this from getting lost: http://github.com/wimleers/fileconveyor/commit/d7faa774d97f1269ba374346e....

Thanks again :)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

tomfilepp’s picture

tomfilepp commented

Pardon my ignorance, but what is a Bucket Prefix? I've never heard of this terminology before with S3, unless you're just referring to s3://?

Apparently can't use file conveyor without this, so wondering what it's referring to.

tomfilepp’s picture

tomfilepp commented

any word on this? it's a simple question, and seems fair considering it's not documented anywhere.

rjbrown99’s picture

rjbrown99 commented

This is a closed issue referencing a supplementary project that is not part of the CDN module. I suggest reviewing the details on the File Conveyor website or github issue queue for it.

http://fileconveyor.org/