[Regression] Add an option for CSS/JS aggregation into a big file each

Latest head, I can confirm this.

Subscribing

see #769226: Optimize JS/CSS aggregation for front-end performance and DX

I just installed a fresh copy of Drupal 7 RC2 and its still broken.

As #6 mentioned there is a long list of css and js files and not one combined file for each.

This is working as intended. The parameters for drupal_add_js and drupal_add_css have changed:

http://api.drupal.org/api/drupal/includes--common.inc/function/drupal_ad...
http://drupal.org/update/modules/6/7#drupal_add_js_options

If there are contributed module files that are not aggregated, and you think they should be (for example extlink.js), then please file a bug with the contrib module involved. It's likely that many modules will need to update their calls to drupal_add_css/js.

According to #998446: Make sure drupal_add_* are in the correct group and have preprocess enabled that is a bug in Drupal. I can also confirm that its not even working for a naked Drupal installation with just Garland enabled.

I can confirm, it's a drupal bug, and is a regression.

While the answer may be difficult to accept, it's right there in the docs: http://api.drupal.org/api/drupal/includes--common.inc/function/drupal_ad... .

More aggregate files are expected in D7. There is no longer one giant file - instead there are 3 groups: CSS_SYSTEM, CSS_DEFAULT, and CSS_THEME. Additionally, within each group you have every_page => TRUE and every_page => FALSE. So there are 6 possible aggregate files. Additionally, different media types (such as print) also result in a different set of aggregates.

I can confirm, CSS/JS aggregation doesn't seem to work with Drupal 7 RC2.

Ok, then works as designed.

@grendzy, so this is correct?

I have 3 js drupal core files.
http://www.infonoticias.eu/sites/default/files/js/js_VF53tELsw3kPql1FJD6...
http://www.infonoticias.eu/sites/default/files/js/js_qAqYycy2_eyLD0joNHI...
http://www.infonoticias.eu/sites/default/files/js/js_b4OM6QJaSijwERFshpl...

I have 6 css drupal core files.
http://www.infonoticias.eu/sites/default/files/css/css_o7uJKH3pyI3Sf8OLz...
http://www.infonoticias.eu/sites/default/files/css/css_2THG1eGiBIizsWFee...
http://www.infonoticias.eu/sites/default/files/css/css_bMb3qiA-ej6GeU1CD...
http://www.infonoticias.eu/sites/default/files/css/css_NSQ8v5VSr6lv0W5Kk...
http://www.infonoticias.eu/sites/default/files/css/css_DKSTEIl2kqY8plIjH...
http://www.infonoticias.eu/sites/default/files/css/css_cleuHFojkzTyTanal...

crosspost

int: Yes, what you posted appears to be correct. 6 is the worst case - sometimes you'll have fewer, maybe 4 or 5, depending mostly on the number of files that have every_page => FALSE. If you want to see more detail about how things are grouped, try installing the devel module and running

dpm(drupal_add_css());

Ok apparently that ... is ... working as intended .. kind of. Another change in D7 i will probably never understand. That might make sense from a core developers point of view, but for me as end user and administrator this simply adds massive I/O on my cache folder, 7 new HTTP requests per page view and more objects for the user to dl ( the parallel download problem ). I can even see some big performance issues with improperly configured ETAGS and Cache Expire headers on misconfigured shared hosts ( those would pop up on D6 as well, but would have way less impact, because there where only two big files ).

I really feel that this is an regression. Can anyone please explain why this change was made?

And maybe we can have an "Please combine all those files into one big file" checkbox under the aggregated options with a helper text that explains what issues that might bring and why you should not enable it. If there are more people feeling that this is needed i'll refill another issue as feature request ( or relabel this one ).

I am also confused, what's the problem with one big CSS and JS file? Why do we need six?

Let's say the css being used on a website is divided in parts A-Z.
All pages need parts A-J, page 1 additionally needs part K and page 2 needs part L:

When you want to have a single css file per page request:

So when the user visits both pages, blocks A-J need to be transferred twice.

By splitting up the aggregated css files, there is one file that contains the blocks A-J. On the first page request, two css files are needed (one containing blocks A-J and one containing block K)
On the second page request, only block L needs to be transferred, as the file containing block A-J is already in the browser, thus speeding up the overall process.

Thats what i thought the benefits would look like from the core dev's perspective. But from the users ( as of enduser and admin ) point of view it looks like :

One big file containing all css of the site

PRO : Only one file in the cache folder per session
PRO : Lesser HDD fragmentation because of only one file
PRO : Only one GZIP process is needed if gzip is enabled ( on the server's, as of the users side )
PRO : The file doesn't suffer from the browsers parallel download problem ( see here or here )
PRO : Only 2 HTTP request for javascript and css
PRO : Only two ETAGS and EXPIRE evaluations needed from the servers and browsers point of view

CON : If some small part in the css changed the user has to redownload the whole file
CON : The initial page visit requires more data to be transfered
CON : A big file may block the the parallel download of components and in the worst case mayorly delay the page load, because its a big clunky file.

Multiple small file containing parts of the site's css

PRO : The initial page load requires less data to be transfered
PRO : If chunks of the css changed only one file needs to be redownloaded

CON : 8+ HTTP requests for CSS and JS
CON : HDD defragmentation and IO load on the webserver because of a large amound of small files created and deleted in the cache folder ( that will be a massive problem for larger websites. Say hi to 8000 files in the cache folder changing every second )
CON : 8+ Gzip/ETAG/EXPIRE operations needed ( depending on how good the cache of the browser and webserver works its more on the client or server side )
CON : If a part of the js/css is significantly bigger than another parts it will block the other files from being processed by the browser. I'm fully expecting this to happen, because 98% of the drupal websites don't have a fake or real CND set up and are limited to the 2 files per schedule rule. )
CON : Will result in even more js/css files, because of wrongly implemented drupal_add_* functions. Thats not Drupal's fault, but the default PREPROCESS = FALSE wasn't a good choice. Its probably 1 out of 1000 cases where a dev that call's drupal_add_* would NOT want his code to be aggregated.

Its probably not a full list and some of the things here might just rely on proper server configuration ( which most shared hosts are not ), but ... i don't see the advantage here.

EDIT

Also see the linked page on #1000472: CSS added with @import are downloaded sequentially on IE, http://code.google.com/intl/de-DE/speed/page-speed/docs/rtt.html#Paralle... and http://code.google.com/intl/de-DE/speed/page-speed/docs/rtt.html#Combine... for more information about the points i've mentioned here. The two pictures of the last link say it all.

You missed my point. In the case where you have only a single css file, for each page where other css is needed, a different file needs to be downloaded, so even when a second page view needs only 5 lines of extra css (which aren't in the css file for the first page because then e.g. that would look bad), the entire css needed for that page needs to be re-downloaded!
When you have multiple files, for the second page view only the additional 5 lines need to be downloaded.

This is definitely by design.

CON : HDD defragmentation and IO load on the webserver because of a large amound of small files created and deleted in the cache folder ( that will be a massive problem for larger websites. Say hi to 8000 files in the cache folder changing every second )
CON : Will result in even more js/css files, because of wrongly implemented drupal_add_* functions. Thats not Drupal's fault, but the default PREPROCESS = FALSE wasn't a good choice. Its probably 1 out of 1000 cases where a dev that call's drupal_add_* would NOT want his code to be aggregated.

On the contrary. The new system can only result in less files cached and smaller ones.

Accepting that the way aggregation works in D7 is "by design" does not mean that it is optimal or a good solution for everyone or a good reason to mark this topic closed when clearly there is much concern around the issue.

It seems to me that the design is based on a use case that is simply not relevant for many drupal users - particularly large site developers and is a real regression from D6. For many sites, putting everything into one big file is optimal - the "a particular page needs 5 lines" argument (#24) doesn't really stand because for most site purposes simply including everything in the one file - even though many parts are not required for a particular page - is optimal where the js/css files are stable. CSS & JS are downloaded & cached once - ALL future css/js demands are now delivered from the browser cache. Of course I also accept that there are other use cases where the current model would be preferred. Eventually in either case the files will all end up in the cache. However - the current design will mean ( whether the file is in the cache or not ) that many more files must be processed and the amount of html to be processed and downloaded IS increased. I must admit I like to keep my html as clean and tight as possible and I don't think I'm alone!

If there are special cases, then it is a straight forward case to simply exclude that code from aggregation, the cost being an additional file..

As D7 stands ySlow is choking on the LACK of full aggregation. I think the idea, mentioned earlier of a checkbox that would allow FULL aggregation of all files marked for aggregation into ONE file is a very positive idea and one that would allow the designers to remain true to their vision whilst preventing a regression from D6 and allowing others who's use cases are different to have FULL aggregation. This is surely not a difficult to implement or controversial proposal?

Just my pennies worth ...

Maybe we should go the route of a feature request then? Since most of us agree that this is an regression, i'll leave it in the title for now.

Great! I think the title change and category is right and properly describes the issue.

I fully expect others who either upgrade from D6 or start new developments with D7 to be looking at this. Maybe enough momentum can be gathered to push for a 7.x target?! ;)

tx

Folks... you need to think about what YSlow is not showing you: how the D6 aggregation strategy impacts subsequent page loads. Once you start adding modules this becomes a real issue, even for small sites.

For a basic example, look at the plus1 module in D6: It adds a CSS and JS file on pages with a voting widget, which will "bust" the aggregate on these pages. The new strategy is a compromise with many factors. Is it optimal for everyone? Of course not, and neither was the old strategy.

The good news is there is room for a contrib module to provide more control:
http://api.drupal.org/api/drupal/modules--system--system.api.php/functio...
It would take about 5 lines of contrib code to revert to the D6 strategy, if that suits your site.

I agree that there will be cases where having the current D7approach may be more practical. But for many people - even in the example @grendzy gave... simply having ALL the css & js in a single file (whether it is used on a particular page or not) is more efficient. If there are modules that do not want to mark their files as suitable to be aggregated for all pages, then the price is additional files. This is a trade-off that should be under the control of the site developer as every situation will be different.

I think there is at least agreement that niether the D6 or D7 approaches are perfect for every situation - therefore the option of a choice seems a sensible way forward. I believe this should be addressed in core not contrib. D8 and either a patch or a backport to D7.x.

There is a Drupal 7 contrib module that does this, I have not reviewed it yet: http://drupal.org/project/core_library

More importantly, I'd like to see javascript aggregation made pluggable in D8, and this backported to D7 if we can do so without breaking bc. I had to do quite unholy things in http://drupal.org/project/agrcache to swap out the js aggregation mechanism.

Regarding #26 - Drupal 6 did not "include everything in one big file", rather it included various combinations of almost everything so that users got a different large file on each page view - this is completely different from a performance point of view. If there was a way to generate a single (or a small number to allow for parallel downloads) file, there would be support for that, but it is simply not possible whilst we have the drupal_add_* based system for adding files (meaning there is no way to "see" all files on the site), and at the same time respecting weights, device classes, IE grouping rules etc.

This has been a dead issue for about 2.5 years.

Also, #352951: Make JS & CSS Preprocessing Pluggable landed in Drupal 8, so in Drupal you will be able to implement this behavior easily.