CDN mapping like this:

http://cdn1.site.com|.jpg .jpeg .png .gif
http://cdn2.site.com|.jpg .jpeg .png .gif

And the CDN server could distribute every second JPG/PNG/GIF from the second CDN.

Alternatively allow path based mapping such as

http://cdn1.site.com|sites/default/themes|.jpg .jpeg .png .gif
http://cdn2.site.com|sites/default/files|.jpg .jpeg .png .gif

Please. :)

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers’s picture

Title: Make it possible to map one filetype to multiple CDN's » Automatically distribute certain filetypes over multiple CDNs
Component: Miscellaneous » Module
Assigned: Unassigned » Wim Leers
Priority: Normal » Major

I'd *swear* I had created an issue for this. I'm 100% certain I've written about this in an e-mail to someone or in some issue here. But I can't find it back.

Yes, I've been wanting to add this. However, I propose a different, clearer syntax, that will also make checking for this more efficient.

I propose this syntax:

http://cdn1.site.com http://cdn2.site.com|.jpg .jpeg .png .gif

It's consistent with what already exists. Thoughts? :)

Wim Leers’s picture

Oh, FWIW, this is already possible currently: see the README. You can even configure it through the UI. It's just more involved than doing this automatically through the CDN mapping field, that's all!

doublejosh’s picture

Perhaps it's not available in the D6 version, but the CDN Mapping instructions are non-existant. It just says "Enter the domain name"

Wim Leers’s picture

You should be seeing a message at the top, telling you to install the Advanced Help module if you want more detailed information. All information used to be in the descriptions of each form item. However, this was so overwhelming that this was actually annoying. Hence the Advanced Help module.

Wim Leers’s picture

Found the post I was referring to in #1: http://drupal.org/node/1410318.

doublejosh’s picture

Quite right. Sorry for the noise.
Am actually interested in folder-designated server selection in the CDN Mapping textarea.

Wim Leers’s picture

Folder-based CDN mapping will not be supported in the CDN mapping textarea. The majority of users don't need this. You're in fact the first user in years to request this.

However, you *can* implement that using the cdn_pick_server() function. Also see the README.

Wim Leers’s picture

Status: Active » Needs review
FileSize
5.49 KB
4.1 KB

Patches for D6 & D7 attached. Please review.

Comes with updated unit tests: #1471472-1: Unit tests! :)

Wim Leers’s picture

FileSize
2.37 KB

Wrong patch for D6 uploaded.

Wim Leers’s picture

When this gets committed, I should also update the README.

doublejosh’s picture

Good to know folder based server picking (for theme vs upload and specific content type upload) will stay in cdn_pick_server().
I personally have had trouble getting that function to actually fire, but I'm sure it's just on my end.

Wim Leers’s picture

I'm afraid I'm sure about that too :) You can always use the UI for that as well on the admin/config/development/cdn/other page.

Wim Leers’s picture

FileSize
4.1 KB

Re-uploading the D7 patch of #8 (my god this is getting messy), so that the testbot will hopefully pick this up :)

Wim Leers’s picture

Wim Leers’s picture

FileSize
4.1 KB

Once more. I've followed the instructions at #1475086-1: Testbot not picking up contributed module patches?.

EDIT: HURRAY!

Status: Needs review » Needs work

The last submitted patch, 1452092-8_D7.patch, failed testing.

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
4.11 KB

Reroll. Should apply now.

Note: because of the testbot, I can no longer upload D6 patches simultaneously, because the testbot doesn't look at filenames, only at the version indicated in the issue. Asked for help in #1475256: Development in 2 branches simultaneously; patches posted for both too: how to make testbot deal with this appropriately?.

Wim Leers’s picture

HURRAY! :D :) :) :)

*Now* we're rolling!

If @doublejosh could now test and review it too, then I can commit this. This would be the first commit for the 2.5 release :)

doublejosh’s picture

#9 is the correct one for D6 yes?
I'm actually just interested in folder splitting, but am working on CDN right now so can do some testing next work week.

Wim Leers’s picture

Yes. Sorry for the mess — not that it's really my fault, but anyway :)

doublejosh’s picture

Ran the patch after updating to 6.x-2.4 with a fail on one portion...

$ patch < 1452092-8_D6.patch
patching file cdn.basic.inc
Hunk #1 FAILED at 96.
1 out of 1 hunk FAILED -- saving rejects to file cdn.basic.inc.rej
patching file cdn.module
Hunk #1 succeeded at 160 (offset -17 lines).
rnyberg’s picture

Thanks for the tips, managed to get the balancing between 2 servers to work so now it's no longer 90% CDN1 and 10% CDN2 due jpegs :)

Wim Leers’s picture

@doublejosh: can you do a reroll of the patch? Recent commits have broken it. The D7 patch should still apply.

Vacilando’s picture

Is the specific file and cdn server link maintained between page loads? E.g. if certain image is uploaded to one of the CDNs, will it be pulled from there also next time, or can it happen that next time it will be uploaded and served from another CDN?
If the link is not maintained, doesn't it mean that all files will eventually be (possibly in different versions) copied in CDN_1-CDN_N (resulting in N x the size of CDN space needed). What's the strategy here?

Vacilando’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
2.8 KB
19.98 KB
24.7 KB

Works flawlessly! See a screenshot of waterfall for one local image subdomain, then two screenshots where images are distributed over 5 image subdomains.

You can see that all images load in parallel, which has been the objective. You can also see that they are called randomly.

I suppose the patch (D7 at least) can now be submitted to dev.

Questions:

  1. See #24.
  2. I wonder why the actual loading time, in the parallel download case, is not much shorter. Seems that the browser is waiting there, so would this be solved by setting Apache's MaxClients higher? Any ideas here?
Wim Leers’s picture

#24: yes, the "link" is maintained.
#25: Great :) The tests say it should work flawlessly, so I'm glad the reality confirms that :) As to why it's not faster: was this the first page load? The CDN (assuming you're using an Origin Pull CDN) has to cache it first, so that may be the reason.

I'll commit this soon then, thanks for the review!

Vacilando’s picture

Cool, looking forward to the new dev!

It's like that for the first and subsequent pageloads. Origin pull - but from a fake CDN (local subdomain) -- perhaps that's the reason. I should open another issue to clarify this because it happens independently from the automatic distribution over CDNs.

Wim Leers’s picture

This happens independently from that. It only depends on the filename you feed it. You can easily see that in the source code too, FYI.

Wim Leers’s picture

Status: Reviewed & tested by the community » Fixed
Wim Leers’s picture

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

steeph’s picture

Thanks for this! It's really simple, like the rest of the module.

I have a view with 10 images on each page. On most pages 3 images get served from the first server and 7 from the second one (on some pages it's 5/5, on some 2/8). Is this intended? It's actually good in my case. I'd like to stay it that way. But I expected about half of the files to be served from each server.

Wim Leers’s picture

#32: this issue was closed a few hours before you commented on it. Hence nobody noticed your comment.

The CDN module does not try to perfectly balance files. It could, but then it could happen that image A is served from CDN X on page 1 and from CDN Y on page 2. Instead of providing perfect balancing, we guarantee that image A is always served from CDN X.
However, if you really would like perfect balancing, then you should open a new issue for that and provide a patch. :)