[meta] CSS and Javascript aggregation improvement

In my opinion, aggregation should remain the more static and simple possible. When I look the actual core, the full mecanism is crap, way too dynamic.

My actual effort if purely optimization and does not provide any kind of API, but you are going this way and it seems good.

Totally agree about CSS/JS aggregation/minification pluggable backend and more API oriented. It should also be easily configurable via settings.php for future sysadmins.

The good thing in my actual implementation is the I/O reduction and the aggressive group merge for CSS and JS aggregation merge. I also provide an IU that can let the site integrator manually choose exactly which file should and should not be aggregated (and I think this is a thing that core should allow).

One last thing, in a future good API, core should force modules to define their CSS and JS files within a hook (such as the library handling) which would allow site admin to do advanced CSS/JS profiles on a per page/per theme/per site basis.

EDIT: If the core could control the full CSS and JS lib list, it wouldn't ever ever again have to do any I/O at runtime.
Re-EDIT: If modules are able to statically declare all their files, an inclusion policy could be configured by integrators, selecting which file(s) should be *always* aggregated wether or not they are being used. Then all others (dynamic includes) should only be included as-is without any aggregation/minification (this is a runtime job but really costs a lot in term of performances).

File existence should never be checked at runtime, modules developers should be aware that a non existing file or a wrong given path IS A BUG and the core graceful downgrade IS NOT A FEATURE but is an ugly failsafe mecanism due to non stricly developped modules.

We totally agree then :)

As I told quickly on IRC this issue open another issue which is the CDN integration. As you use the same mecanism (menu callback) to get the static CSS and JS files, this is the issue with the image handling mecanism and this mecanism could even be generalized (it would gracefully go in the file stream handling way of doing things).

EDIT: CSS and JS could declare schemes such as css:// and js:// and then benefit from the stream wrapper interfaces and still be using their custom implemention, therefore reducing greatly the full codebase.

The menu callback for accessing file should be generalized to all files, files would then be statically served if exists, or would call a stream wrapper interface method (may be such as generateFile() or such), therefore call the scheme specific implementation behind.

When it comes to #1, how are you taking the hash and deriving the files that should be included in the response? The idea of doing the JS/CSS in this form is something a number of people have been talking about for some time. It's nice to see a place to start discussing that idea.

Pluggable JS was started over on #352951: Make JS & CSS Preprocessing Pluggable. It stopped waiting for a common plugin mechanism that never ended up materializing in D7. The same concept could apply to CSS.

I was thinking a little further on #1. If we take a look at what espn.go.com does we see something like:

<link rel="stylesheet" charset="utf-8" media="screen" href="http://a.espncdn.com/combiner/c?v=201010121210&css=global_reset.r1.css,base.r228.css,modules.r424.css,global_header.r38.css,modules/global_nav.r47.css,/espn/espn/styles/frontpage_scoreboard_10,modules/insider_enhanced.200910131831.css,sn_icon_sprite.200907150955.css,mem2010/mem.r5.css,mem2010/mem_espn360.r2.css,universal_overlay/universal_overlay.css,universal_overlay/media_overlay.css,universal_overlay/video_overlay.css,universal_overlay/photo_overlay.css,universal_overlay/dyk_overlay.css,warvertical12a.css,twin.css" /> 

...

<script src="http://a.espncdn.com/combiner/c?v=201012011221&js=jquery-1.4.2.1.js,plugins/json2.r3.js,plugins/teacrypt.js,plugins/jquery.metadata.js,plugins/jquery.bgiframe.js,plugins/jquery.easing.1.3.js,plugins/jquery.hoverIntent.js,plugins/jquery.jcarousel.js,plugins/jquery.tinysort.r3.js,plugins/jquery.vticker.1.3.1.js,plugins/jquery.pubsub.r5.js,ui/1.8.2/jquery.ui.core.js,ui/1.8.2/jquery.ui.widget.js,ui/1.8.2/jquery.ui.tabs.js,plugins/ba-debug-0.4.js,espn.l10n.r8.js,swfobject/2.2/swfobject.js,flashObjWrapper.r7.js,plugins/jquery.colorbox.1.3.14.js,plugins/jquery.ba-postmessage.js,espn.core.duo.r50.js,espn.mem.r15.js,stub.search.r3.js,espn.nav.mega.r24.js,espn.storage.r6.js,espn.p13n.r9.js,espn.video.r33a.js,registration/staticLogin.r10-14.js,espn.universal.overlay.r1.1.js,espn.insider.r5.js,espn.espn360.stub.r9.js,espn.myHeadlines.stub.r12.js,espn.myfaves.stub.r3.js,espn.scoreboard.r4.js,%2Fforesee_v3%2Fforesee-alive.js&debug=false"></script> 

We could do something like:

script src="http://example.com/files/js/js_hash12342341234234?files=misc/jquery.js,misc/jquery.once.js,misc/drupal.js"></script> 

The hash would be built like it is now and the combiner could know the files to use. This could be easily cached by the browser and we could test to make sure the files generate the right hash for security. I even like that it documents what files are included in the hash as it can be a pain to discover when you do need to know.

I'm against showing the real file names in the URL. The combiner shouldn't get its parameters from a GET request it's unsafe. It means anyone could DDoS your site by doing this request and putting any parameter. It could even lead to security issues where people could ask for some other arbitrary files to be aggregated in the final files.
It should remain a generated hash that exists in a variable somewhere server side.

@pounard You can do it in a safe and secure manner.

I agree that avoiding file stats at all costs is important.

Without being involved in the latest initiatives regarding front-end performance, I apologize in advance for not knowing about current patterns being applied. But, I do have the following question:

Far Future Expires headers are important to maximize client-side caching of files. Far Future Expires headers require unique file URLs however. How to map a file URL to a unique file URL in an efficient manner? (And: file URLs should change whenever their contents change.)

I know this does not directly affect core (yet), but it's an important best-practice pattern to be agreed upon and then documented.

CDN module

I'm currently working on a patch for the CDN module that adds this: #974350: Far Future setting for Origin Pull mode. However, it focuses only on adding Far Future Expires functionality, it does not yet avoid file stats. Essentially, it does two things:
1) Detect the last modification time of a file and use this to generate a unique file URL:

+    // Alter the file path when using Origin Pull mode and using that mode's
+    // Far Future setting.
+    if ($mode == CDN_MODE_BASIC && $farfuture) {
+      $path_before_farfuture = $path;
+      $path = 'cdn/farfuture/' . filemtime($path) . '/' . $path;
+    }

2) Then there is a menu callback defined at cdn/farfuture (meaning that any additional arguments will be passed to the menu callback function). This function then exhibits all the most optimal behavior: optimized caching headers for file URLs, gzip support, range request support (see cdn_basic_farfuture_download() in http://drupal.org/files/issues/974350-12.patch) and correct replies for If-Modified-Since requests.
I know, it's silly to implement all of this in PHP instead of at the web server level. But that is the only realistic solution to guarantee this to be working: cheaper hosting accounts don't allow for all required functionality to be defined through .htaccess files anyway. In the future, I can still provide a .htaccess file that expert users can enable (yes, the goal is to make the CDN module usable for everybody; and yes, CDN's are affordable already).
The file_exists() calls in this function don't matter much, since they'll only be called very rarely, and only by the CDN.

Summary

So, in summary:
1) I have a working solution for unique file URLs, to allow CDNs to be used with Far Future Expires headers
2) this solution does not work without CDNs for large sites: the load would grow too fast due to the file_exists() calls
3) for very large sites, it still doesn't work quite well because of the filemtime() call

It is point 3 in the summary that is problematic and for which I'd like to see a best-practice pattern emerge.

Thoughts?

if your not opposed to 2 database tables & one cache table you could use what I got in here.
http://drupal.org/project/advagg
does imagecache generation.
uses zero stat calls when serving a page; even with imagecache style generation off.
hooks for implementing things like jquery cdn, csstidy, jsminplus.
gzip (but this is in D7 now).
uses locking to prevent multiple threads from doing the same thing.

Looking advagg's issue queue, there are still a lot of issues to workout, but the core part of this is working correctly.

@#14 Thanks for sharing.

Hitting the good ol' non-existent subscribe button.

@#16 Agree about a lot of things, especially with having a more flexible grouping logic.

I agree with everything outlined in #16. The only problem is: time. I won't have time until this summer at least (i.e. after this semester, after my master thesis). If we get this right, Drupal will lead in the WPO realm.

Also see http://drupal.org/project/wpo.

Subscribe.

Subscribing.

From the summary 2 and 3 is what #1751602: Use Assetic to package JS and CSS files is aimed at solving.

Assetic can help with bundling, http://symfony.com/doc/2.0/cookbook/assetic/asset_management.html#combin... and since Assetic is pluggable I'm pretty sure we can work something out. For example I just ran into this: https://github.com/smoya/AssetManagementBundle, I wouldn't rule it out just yet.

If you preprocess stuff, you need to dump the files somewhere, which partially fixes #2 if you make the preprocess step mandatory. Concatenation is enough preprocessing for that.

EDIT: Yes the second link adds logic we would need on our side in order to achieve it fully, I tested the equivalent for ZF2 a few monthes ago.

#352951: Make JS & CSS Preprocessing Pluggable landed today.

Having reread this issue, #16 captures the gist of what needs to happen. Thanks to #352951, we can now:

• plug in JS minification (contrib can do this, core can add it without changing APIs): override asset.js.optimizer
• plug in different grouping logic: override asset.(css|js).collection_grouper
• plug in different aggregate expiration logic: override asset.(css|js).collection_optimizer
• plug in a script loader : override asset.js.collection_renderer

So, it is now a matter of deciding what we still want to do for Drupal 8 core. I think removing file_exists() calls is definitely a good idea. It can be achieved by fixing our current CssCollectionOptimizer and JsCollectionOptimizer: it won't break D8 APIs, so we can still do it. Shall we get this done over at #1014086: Stampedes and cold cache performance issues with css/js aggregation?

What else should we do for Drupal 8?

P.S.: inlining of background images in CSS files is now possible using hook_file_url_alter() thanks to #1961340: CSS aggregation breaks file URL rewriting because it abuses it.

.

This one can be closed, no?