Our current drupal_build_css_cache() and drupal_build_js_cache() logic works like this:
* There's a request to the page.
* If the bundle doesn't already exists, drupal_build_css/js_cache() will generate and save all it.
* After this initial request is finished, the browser the requests the saved files as normal and serves the page.
There are multiple issues with this, most of which is documented at #886488: Add stampede protection for css and js aggregation - to summarize:
* There is no logic in HEAD to prevent race conditions where the same file could be built multiple times by different requests to PHP, if you have numerous bundles and high traffic this can cause i/o issues.
* On an i/o constrained setup, I'm seeing this process take around 400ms (for ten bundles in total between css and js), that's around 30% of the entire request as measured by webgrind.
* Attempts to introduce locking can lock every PHP request to the site - for example if a bundle is served on a large number of pages that all get requested during a cold start situation, and they get stuck in lock_wait() for the first request to build and save the file, so while doing this can reduce the i/o impact, it can still take a site down due to hanging apache processes.
* Every bundle saved is also a call to variable_set(), which has its own issues with locking and race conditions - see #973436: variable_set() should rebuild the variable cache, not just clear it and #987768: Optimize variable caching for attempts to solve this.
So I'm thinking we should look instead at copying what imagecache does:
* Register menu callbacks for files/css and files/js
* Make sure it's possible to reverse-lookup aggregates by filename - we'll need to have a record of the hash stored alongside the files and their order used to created it.
* Normal page requests will only confirm that we have the hash stored (probably in variables), create it if it's not there, and generate the link to the aggregate.
* The callback will pick up the filename, generate the css/js aggregate, and serve it from PHP the first time it's requested.
* In the case of stampede, we can check file_exists() both before building the file, and before trying to actually save it. As soon as the file is saved, subsequent requests won't be hitting PHP anyway ( afaik image module doesn't do this at the moment, but it probably could)
No code yet, just getting issue the issue written up for now.