Several related issues filed together -

1) Remove core dependency on CDN module
Right now all static file requests go through cdn_file function. This is a problem because a patch to the core depends on a module-specific function. If we remove the dependency (by using a configurable static_file function), support for CDNs could become part of core and we could allow multiple CDN implementations, as well as local static file servers (such as Lighttpd). I posted a full implementation for D5 patch with JS aggregation (http://drupal.org/node/149402#comment-699065) and a D6 issue (http://drupal.org/node/212369).

2) Fallback on a local static fileserver
If cdn_file cannot find the file remotely it serves it from a local server. The only problem is that if CDN is out of sync the requests for static content would go to the Apache server. A possiblity of this happening would make it less practical to set KeepAlive to Off. A solution would be have a setting allowing to serve static files from a dedicated local server. This is supported by the patch from the Item 1 above, in case CDN is not enabled. CDN module should support local static file servers as well.

3) Implement Cron hook
Requiring a separate cron job for cdn_cron.php complicates management and administration of the module. CDN module should implement a cron hook, like everyone else. There could be an admin setting that could disable invokation of the actual sync job for sites that prefer to use the standalone script.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers’s picture

Thanks a lot for your feedback! :)

1) Will look into this ASAP.
2) I planned that as a future feature. Will look into this as well.
3) I disagree here. CDN cron typically takes a lot of time (relatively), and for that reason, it deserves a separate cron run, so it doesn't slow down the normal cron.

(I preferred to reply faster and less complete. Will answer #1 and #2 later today.)

dkruglyak’s picture

Thanks for such a quick follow-up :)

Items #1 and #2 are very much related. They can be solved together with the use of my code from D5 thread. We still have a chance to get proper CDN support into D6 core - please review and comment on the issue I filed above.

On #3 I agree with you, however some installations use custom balancing of cron jobs - but need to ID them by hooks. I suggest to implement the hook, but run the job subject to module's admin setting. Does not break your use case.

Wim Leers’s picture

#3: Excellent idea! Implemented it right away, see my latest CVS commit.

Wim Leers’s picture

#1: static_file() is a bad name. I'd opt for file_url(), which is much more descriptive. Other than that, I agree.

#2: you're not quite right here, I think. You're saying that your patch from #1 has the ability to try multiple servers, i.e.:

  • File exists on CDN? Serve from CDN and stop, else continue
  • File exists on static file server? Serve from static file server and stop, else continue
  • Serve from Drupal's web server

I agree that this is a necessity, but your patch doesn't implement this yet as far as I can tell.

dkruglyak’s picture

My comments posted here: http://drupal.org/node/149402#comment-700138

Perhaps the name should be static_file_url, since this is not for just any file but only static content.

Feel free to propose a different implementation, but consistent with criterias I posted in the other issue.

Wim Leers’s picture

Files are static content.

I've already replied to your comment, too.

dkruglyak’s picture

Status: Needs review » Active

OK, I do not really care what API we have as long as it works, supports static fileserver without CDN and could get commited to D6 core. I suggest we use this thread for D5 patch development, instead of JS aggregator issue.

Your patches look OK, but a missing piece is a default function with constant static URL. You should add it to common.inc:

function static_file_url($file_path) {
  $prefix = variable_get('static_url', base_path());
  return $prefix . $file_path;
}
dkruglyak’s picture

Status: Active » Needs review

With more thought, I think it would be nice to make CDN module the main place for distributing D5 JS aggregator patch. I compared your code in CDN module with the one I am running and here are several issues with your JS aggregation patch:

drupal_add_js
You are adding compressed JS into header, rather than footer. Could this cause problems with contrib modules?

drupal_build_js_cache
Why remove the compression, what is the problem with it? Even if you do not like it why not make an optional setting and let admins decide?

system_performance_settings
#default_value values need to be wrapped in intval to convert non-empty FALSE values. Packer setting can be added here too.

Wim Leers’s picture

Status: Active » Needs work

Those aren't issues.

- JS in header delays the rendering of the page. I'll be trying to get D7 to put it in the footer as well. Until somebody
- Compression has been *removed* from Drupal 6 core because it's so problematic. The only compressors that don't result in problems at the moment are the YUI compressor and Dojo Shrinksafe, but both require the Rhine Javascript engine, which is written in Java, so we can't depend on it for Drupal core.
- system_performance_settings: I didn't write this, I didn't modify this part of the D5 backport of the JS aggregation patch.

I'll add static_file_url().

dkruglyak’s picture

1) OK, I read the D6 thread about removal of the packer and seems like this was a very controversial decision. But I'm fine with following D6 here. Looks like packer will have to be a separate patch anyway.

2) If you are moving compressed JS to footer, you have to make the change in every contributed module (even in this patch). How are you going to do that? If you are following D6 design on packer you should not break header/footer placement. Make a separate patch for that.

3) As for system_performance_settings: take a look how intval was added for CSS setting in 5.5 system.module, but left out of the patch.

Wim Leers’s picture

1) It /seems/ controversial, but it's a solid decision. It just breaks JS in a lot of cases. Even Drupal core's jquery.js file, which is compressed with the packer, is often problematic.

2) Incorrect. 95% of all modules don't set the location argument. Simply changing the default will thus affect all of them. This has nothing to do with JS compression.
In my personal experience, and I use a fair amount of modules that ship with JS, it works for all of them. :)
And making it a separate patch makes sense, but then you'd have to apply two patches. One for file URL rewriting + JS aggregation, and one for changing the JS location from header to footer. That's why I opted for a single patch. The added benefit is that people actually *test* if it (i.e. JS location) breaks any JS out there, so I can choose to change my strategy.

3) I didn't notice this had changed in 5.5. The JS aggregation backport patch was initially against 5.1, so that's why I guess. :) Will fix!

dkruglyak’s picture

Even your own patch still uses 'header' in update.php and drupal_get_js function.

Contrib modules assume all JS goes into header, so moving it into footer is unsafe unless you are going to audit/debug all contribs. I did a quick grep and looks like quite a few modules use header explicitly (eyedrop, tinymce, nodeformpopup, oaliquid, oainjection, jrating, kudos, google_analytics). The list is surely incomplete, I do not have all modules installed.

Please make a separate patch for footers so anyone who really wants to debug them can choose to do so.

Wim Leers’s picture

update.php is only used by admins, so optimizations there are unnecessary, which means I did this by intention.

The scope musn't change in drupal_get_js(), but in drupal_add_js(), in which it *is* changed.

I know it's unsure what the results will be, but I couldn't produce any issues. I guess I should keep drupal.js and jquery.js in the header, for the sake of compatibility with modules that add it to header. But at least the google_analytics can (and even should be) put in the footer. Most people add it to the header out of habit.

Wim Leers’s picture

Title: [Multiple Enhancements] Core patches, local content servers and sync job management » Core patch improvements
Status: Needs work » Needs review
FileSize
80.23 KB

Not actually a patch, but a screenshot that's the result of my patch.

I'm rewriting the patch once again, to provide an UI to configure the file servers (that's what I decided to call them in the end).

Things I've learned:
- Provide 2 new hooks instead of hacky $conf array entries: hook_file_server() and hook_file_server_info.
- "file server" is the best thing I can come up with, and the easiest to understand for the average Drupal user. Feel free to make suggestions.
- The static file server shouldn't be supported by Drupal core. It'd be too advanced for the average Drupal user, as illustrated by the attached screenshot.

More patches will follow later. I'd like to get some initial feedback first.

Wim Leers’s picture

FileSize
112.02 KB

Improved UI:
- Better text.
- No more static file server in Drupal core by default.
- The default file server - the web server itself - is now always displayed as the last file server, i.e. the server that will always be fallen back to.

dkruglyak’s picture

Assigned: Unassigned » Wim Leers
Priority: Normal » Critical

There is no reason to exclude static file server from Drupal core. This is just two lines of code that does not need overhead of a new module.

Wim Leers’s picture

There's a very good reason actually IMO: usability. A large part of Drupal users don't know what a static file server is, or at least shouldn't be bothered with it.

Wim Leers’s picture

Another reason is that you may want to write the static file server module in such a way that you can use *several* of them, choosing by filetype (e.g. streaming video versus images) which server in an array of servers should be synced too/served from.

dkruglyak’s picture

This screen is for site admins who should know what a file server is. Two lines of explanation is all that is needed.

To choose by filetype, sure there should be a separate module. But a catch-all static file server is a very trivial function, suited for core.

Wim Leers’s picture

FileSize
11.58 KB

Well, then I will leave it out for the sake of simplicity, so I can get it in Drupal core faster. Static file server support could still be an easy follow-up patch.

Attached is the updated patch I just committed. It provides a single new hook: hook_file_server().

dkruglyak’s picture

Would extra two lines make it harder to get in the core? If this has to wait till D7 there should be plenty of time to review and make further improvements to file serving. If on the other hand we could actually get any kind of patch into D6, I would be happy with anything.

At least static file serving should be available with this CDN module either within the core patch or the module itself, for D5 and D6.

Wim Leers’s picture

I want to keep the core patch as simple as possible - I've said this before. It's not just two lines. It's a new feature, with usability consequences and so forth. I won't change my mind, so please stop asking for it.

And I will most definitely not include the static file serving feature in the CDN integration module. That's completely beyond its scope.

I have implemented everything you've asked for in this issue. Please come up with *constructive* feedback about the current core patch.

dkruglyak’s picture

Constructive feedback is that CDN should fall back on static file server, not Drupal's Apache that would cause problems with KeepAlive.

Why create yet another module for (yes) just two lines of code that replace base_path() with a variable? This makes no sense at all.

Wim Leers’s picture

Status: Needs review » Fixed

1) It's not cdn module's task to provide a fallback mechanism. It's Drupal core's. And that's what the core patch allows you to do. So this is already solved.
2) I've explained it multiple times already now. I won't repeat myself again.

Marking as fixed, since the original requests have all been implemented.

dkruglyak’s picture

Status: Fixed » Active

Apparently you do not want to hear about the problems with your design. No, you did not fix the original item (#2).

In cdn.inc file the function cdn_file_url returns base_path() if the file is not found on CDN. This behavior is a problem. If the file is not found on CDN I do not want to return base_path(), I want to fall back on a local static file server. Otherwise there is a performance penalty for turning off KeepAlive on Apache, which is the whole point of sending static content somewhere else.

I do not have time to argue with you further, but perhaps you might read Robert Douglass' article to understand how to optimize performance of static file serving: http://www.lullabot.com/articles/using-lighttpd-static-file-server

Wim Leers’s picture

Status: Active » Fixed

Apparently you're incapable of reading what I write. I've repeatedly said that the design is that the file_url() function calls a module's hook implementation (e.g. the CDN module's), and if FALSE instead of an URL is returned, then it will try the next server.

See the code from the patch:

// Try the first preferred file server to check if it can serve the file.
// Then try the second, and so on.
foreach ($modules as $module) {
  $file_url = module_invoke($module, 'url', $file_path, $absolute_url);

  // If the URL rewriting function sucesfully generated a URL, we can stop
  // trying to find a server that can serve the file.
  if ($file_url) {
    break;
  }
}

I just discovered that somehow, I failed to commit the change to cdn_file_url(). Why couldn't you just point me to that, instead of saying that the design has problems?

Wim Leers’s picture

Created an issue in the issue queue of the Drupal project: http://drupal.org/node/214934.

Anonymous’s picture

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.