Gzip aggregated CSS and JS

I just noticed a typo in the benchmark tables - 'Faster By' for HEAD should of course be 0% (or N/A!), not 100%.

Subscribing... this should handle CSS and JS files...

With 4 weeks to go, let's see if we can get this in :-)

Owen, did you want to start with a patch based off of your module? If not, I can try and post one in the next few days.

Subscribing.

subscribing...

This had huge potential but unfortunately didn't make it into 6.x. What's holding up the progress on this patch? Is it because of the outstanding bugs on gzipped page cache in core?

Also I agree with Grugnog2 in #5 about the use of .htaccess method, while it is browser compatible, Dries highlighted his desire to make core compatible with lighttpd and other web servers. We should put some consideration for those too if possible.

subscribing

Got tired of waiting and just added it to my installation. Only requires 14 extra lines in .htaccess and 2 in common.inc. These 2 lines need to be added again after updating, but imo it's totally worth the hassle.

The step by step guide is over here:
How to GZip Drupal 6.x's aggregated CSS and JS files

Surprisingly you're correct. It's silly, but there are indeed a few installations without Zlib support.

If it's implemented as an option, it would be a good idea to guard the setting (defaults to off) with a function_exists('gzencode') condition.

Fortunately the level parameter for gzencode exists since 4.2 and Drupal requires 4.3.5+. So, at least that part won't be an issue.

subscribe.

Here is a patch, based somewhat on the original patch I posted, but with improvements based on http://kaioa.com/node/78 and http://drupal.org/node/290280

Here is how this should work
* It should fail safe to the regular uncompressed aggregated CSS or JS in almost every situation. This includes gzencode not being available (e.g. on very cut down windows PHP builds), if mod_rewrite is not available and enabled, if the gz file is not found or the browser does not accept gzip encoded data.
* It sends both the correct file extension (i.e. we don't ask the browser to request a .gz file, but just rewrite the request behind the scenes) and also the correct MIME type for the file.
* It does not involve any kind of PHP bootstrap at all - everything is done at the Apache level. In addition, the .htaccess files are created dynamically within the files/X directories, meaning that we don't invoke additional rewrite rule tests (a concern of Moshe in #3) or clutter up an already complex /.htaccess.
* There is no additional user interface cruft. We don't have options for core page level gzipping and I don't see any need for them here, as long as we give it sufficient testing cross-platform. This is a no-brainer.

I have tested this on my system and it worked very well, on the first run in fact (the general approach is also quite battle hardened on many production sites).

If you want to test this out you can do this by applying the patch and then checking the response headers using the "Net" tab in Firebug or a similar tool for both .css and .js files. You should see "Content-Encoding gzip" in there and also notice a noticeable drop in file size.

If someone would like to write some tests to polish this off that would be a valuable contribution.

@Rob Loach

We are testing function_exists with gzencode for the same reason we do this with the core page gzipping code. Zlib support for PHP is extremely common, but is actually not enabled in a default vanilla compile of PHP (even in 5.2), so it is quite possible that it will not be available - "Zlib support in PHP is not enabled by default. You will need to configure PHP --with-zlib[=DIR]" from http://www.php.net/manual/en/zlib.installation.php

If you have a chance to do some tests that would be awesome!

The cache clearing functions do not remove the .htaccess file - all dot files are excluded. This shouldn't matter either way, since the file would be recreated anyway and is harmless if no .gz files are present. Possibly we might want to add a line to delete the .htaccess file at the point when aggregation is disabled (although it causes no issue if it is there), or perhaps even the js/css directory in it's entirety (on the basis that it is cruft and could be confusing to a newbie).

subscribing

I think this should be solely be managed through contrib once #352951: Make JS & CSS Preprocessing Pluggable is in.

Update: Contributed modules can now do this for 6.x. Once the code freeze is in place (early September), porting these to 7.x shouldn't be that hard.
http://drupal.org/project/javascript_aggregator
http://drupal.org/project/css_gzip

Add tag.

#290280: Add GZip compression

khm.

#35 doesn't work in IIS.

@hass IIS rewrite rules are possible different

@mikeytown2: IIS does not have any GZIP compression feature build in for dynamic files (only for static HTML files). You need to buy extra plugins to archive this - if you think you really need it...

Gzip setting controlled by page_compression. Trying to make this as simple as possible.

The attached patch includes the same conditional checks as page gzipping, so should fail where php has no zlib. The .htaccess will only trigger for .gz files exist, so nothing should break there.

The only thing I was wondering is if the rewrite rules for this should also be within a block (which could also contain the mime rule) - because if rewrite is enabled but mime isn't it seems there is a possibility of the type being incorrect, right?

Due to apache weirdness I'm setting the type twice. Once globally in the FilesMatch argument; the other time with T=text/javascript in the RewriteRule. That seems to work 100% of the time on various versions of apache. Does IIS work with T= ?

hass, good to hear that works. Here's a slight adjustment to #46, adjusted the regular expression for the rewrite rule so it compares from the start, instead of from the end. Should always work with *.css?5 and things like that. The reason I need both T=MIME-type and the FilesMatch argument is because the newest version of apache doesn't work with T=MIME-type; well you can get it to work if you do some ugly tricks, so FilesMatch is the best way to get around the apache inconsistencies.
http://httpd.apache.org/docs/trunk/mod/mod_rewrite.html#rewriterule

'type|T=MIME-type' (force MIME type)
Force the MIME-type of the target file to be MIME-type. This can be used to set up the content-type based on some conditions. If used in per-directory context, use only - (dash) as the substitution, otherwise the MIME-type set with this flag is lost due to an internal re-processing.

http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewriterule

'type|T=MIME-type' (force MIME type)
Force the MIME-type of the target file to be MIME-type. This can be used to set up the content-type based on some conditions. For example, the following snippet allows .php files to be displayed by mod_php if they are called with the .phps extension:

Another reason why this should be in core
http://code.google.com/speed/articles/gzip.html

Above patch should still be good... re-submitting it.

Suppose bot was broken, so re-test

@meba
if the .htaccess file is not changed, then nothing happens; gzipped files do not get served. Performance impact is minimal; I haven't done any benchmarking though; It's hard for me to get access to an idle computer (my benchmarking results would be very unreliable for this small of a change).

I don't understand your question, but here's an answer that hopefully answers what you where trying to say.

The new rules look to see if a css.gz file exists; if it does, that gets sent to the browser. If aggregation is turned off, and you compressed the css files your self, it will serve those as well (css.gz files in themes, ect...). If aggregation is on, printing styles get aggregated; thus they get gzip compressed.

Your proposal would make the page cache more complicated. it would require 2 different html sets; one with style.css, another with style.css.gz. It would also be incompatible with older webkit browsers (Apple Safari).

Have one of you verified this in IIS with Helicons ISAPI Rewrite? We are trying to do the same in our non Drupal app and it's not working.

May be related to missing IIS support of:

We tried to do the same in our ColdFusion application on IIS6, but we haven't found a solution yet. I haven't found the time to test the above patch, but I expect after our tests this will result in a broken Drupal site (the gzip'ed CSS/JS files are send to the browser without encoding gzip).

The ISAPI_Rewrite solution from Helicon-Tech do not have the ability to send responds headers. Helicon "Ape" seems to have such a feature, but is only implemented well on IIS7, plus I do not have "Ape". Not sure if there are other ISAPI add-ons for IIS available with such a functionality...

How should this help?

This needs to be togglable separate from page compression. People might want to use different methods for compression of page requisites/support files (such as mod_deflate).

@hass
Have you looked at The Microsoft URL Rewrite Module for IIS 7.0

@sun
Good point. I was kind of hoping that #459980: Rework admin/settings/performance would be in by now... anyway here is the patch with more fine-grained control.

"CSS" and "JS" should be written all-upper-case here.

I see a conversion from .* to .*.gz, but where are those (page) requests handled?

Oh my - with this patch applied, we have this (complex) check at several places throughout core.

At the very least, we want to add a comment what is being checked here and more importantly, why.

Ideally, though, we want to create a helper function that performs those conditions centrally and just returns the result.

This means I can't enable bandwith optimizations while enabling preprocessing optimizations? That's ugly UX.

23 days to code freeze. Better review yourself.

As far as I get it: #disabled _really_ only is when: (function_exists('gzencode') && zlib_get_coding_type() == FALSE) === TRUE

As far as the form is concerned, I'd just go with above condition for #disabled. And let the UX team figure out how to handle dependent configuration settings like this - in a separate issue. (please)

However, I still don't get how *.css.gz or *.js.gz files are handled on the server-side, based on the rewrite rules applied here. Perhaps, I'm just dumb, but it seems like *.css is rewritten to *.css.gz, and Apache will expect those files already exist, while I'm too blind to see a condition that checks whether they really exist?

Or is it the -s option?

All we could do about those aforementioned compression conditions is to create a new drupal_server_compression_enabled() function. Not sure whether that makes ultimately sense, and most probably, this should be deferred to a separate issue... :-/

For binary choices I would go for simpler checkbox:

#459980: Rework admin/settings/performance went in, so I raised up some issues with it.

Here's the patch again. After encountering a misconfigured server on the boost forum, I did some research and almost no one sends out x-gzip anymore, so i switched it to gzip going against the apache documentation (ie 5.5 sends out gzip, so I think it's stuff like Netscape 3 that uses x-gzip).

The last submitted patch failed testing.

aggergated -> aggregated

Updated patch to move the various rules into a single block, and ensure that they are all dependent on BOTH mod_rewrite and mod_mime being enabled (the previous patch broke if you had mod_rewrite enabled, mod_mime disabled, which while unusual is not impossible). I tried to improve the comments as well, to make some of the nuances mentioned in this thread clearer - I think it is certainly easier to see what is going on when the rules are in a single block. We could probably be even more detailed here if we wanted, but I don't want to make it unnecessarily verbose.

I left the settings in for now, but I think these are mostly unnecessary (I can't reproduce any problems even if I have AddOutputFilterByType DEFLATE in my .htaccess) and if we can add a check to see if the webserver (or a proxy) is already gzipping files, as suggested in #459980: Rework admin/settings/performance I think all these settings could be removed.

+1 for Owen Barton's patch

+1 to #87 - I see no need in setting, only if someone get a trouble with double compression can uncomment line in settings.php

So better to make a line with comment in default.settings.php

.

tagging (there are many topics about css aggregation)

sinjab requested that failed test be re-tested.

This is not API change, so we can still hope #87 commited

Here is an updated patch - the htaccess rules are unchanged.

This includes 2 tweaks to the test for when we should save gz files - the first is that it uses the updated "extension_loaded('zlib')" test for php gzip support (the same check we now use for page cache gzipping), the second is that we only save gzip versions if clean urls are enabled. This is because on some hosting setups the new rewrite rules could 404 on css and js if the RewriteBase needs to be set, and isn't - this way we get the added safety of the clean url tests, yet can still support this by default on the majority of sites.

I also included a block in the default.settings.php that describes what is happening, and allows the user to disable it if they choose. The only examples I could think of for disabling this would be if you have an external caching reverse proxy and want to unload gzipping to that box, or if you are using another webserver, and want to save the (tiny) load of gzipping files that will never be served.

It is already :)

I agree that we have to send proper Vary headers.

Will "for newer Apache" hold true in 2 years from now?

*the

114 critical left. Go review some!

There's a typo (twice) in the help text:

"and he zlib extension"

The last submitted patch, drupal-101227-109-css-js-gzip.patch, failed testing.

A quick benchmark comparison with mod_deflate shows this method to be roughly 3X faster.

On the downside, I couldn't get this patch to actually run (I had to manually create the .gz file).

I got the following notices: (odd though that the test bot didn't report any)

looks like some white space issues got in recently... anyway moved the logic to the correct location
http://api.drupal.org/api/function/drupal_build_css_cache/7
http://api.drupal.org/api/function/drupal_build_js_cache/7

I have tested this locally and verified that the code works

That's really great, the size is reduced to more than half. Thanks owen for the benchmarks

@mikeytown2 - I think we are close. Could you take a look at this one?

I have tried to streamline the rules at bit, which should make it simpler and more robust (and a tad faster). As far as I can tell from my tests, functionality is unchanged. It is now using mod_rewrite for both the rewrites and setting the content type. I found this worked fine as long as the content-type is applied as a separate rule. The issue with T= in Apache trunk appears to have been fixed, AFAICT. I also switched to using mod_headers to add the Content-Encoding: gzip header.

Between the 2 of these changes this means we can drop the requirement for mod_mime (not that this was an issue, but it does simplify the rules a fair bit). Also, I think we need to wrap the entire block in IfModule mod_headers.c - the way the previous patch disabled all css/js caching seems much worse to me than getting non-gzip content but being allowed to cache. Finally, I added a snippet to INSTALL.txt to note that mod_headers needs to be enabled for this functionality to work - we should add a link to a supplemental documentation page once this goes in.

I have tested this on Ubuntu 10.04 and an older Redhat based server, as well as with the following versions of Apache:
2.0.63
2.2.4
2.2.14
2.3.6 (released 2010-6-18)

In terms of mod_headers, I agree using it may reduce the reach of this patch somewhat. However given that mod_headers is pretty much the only way (AFAICT) to inject the Vary Accept-Encoding header it seems like we don't have a lot of choice, and so we might as well use it for the Content-Encoding header also. I did have a quick look at using mod_negotiation instead of these rules, however I think we would need to save 2 copies of the uncompressed filed with content type extensions (i.e. .js.js), and it was unclear if mod_negotiation would produce correct output (including the vary header).

I didn't test this on Apache 1.3, as it wasn't compiling properly on my Ubuntu box. Apache 1.3 does claim to support the T= flag, so in theory it should work. I don't have a GoDaddy account, but it does sound like a good place to test.

So I figured out a neat fix for mod_deflate double gzipping content - we can add "E=no-gzip:1" to the content type rewrite rules. mod_deflate (if present & enabled) picks up the no-gzip environment variable and stops gzipping this request. The E= flag is present in Apache 1, and has the benefit that we don't need to depend on and check for mod_env/envif.

Also, while the major benefit of this patch is clearly for end users (see #115) I thought it would be fun to give it a quick server side benchmark too:
Without patch, no mod_deflate: 102007 bytes/req, 1232.03 req/sec, 0.812ms mean
Without patch, with mod_deflate: 33742 bytes/req, 110.56 req/sec, 9.045ms mean
With patch, no mod_deflate: 33675 bytes/req, 1371.40 req/sec, 0.729ms mean

As you can see, mod_deflate enabled creates over 10x more work than mod_deflate disabled (either with or without the patch). With the patch is a tad less server work than without, but it was not that significant in these tests. However we should bear in mind that these tests were over localhost loopback, and in reality there would probably be some scaling benefits in Apache being able to hand off the smaller gzip responses faster at client connection speeds, and hence sustain a higher request rate.

Finally, I managed to install Apache 1.3 with some old Debian packages, and can confirm that it works correctly with mod_headers both enabled (gzip content, headers are correct) and disabled (serves non-gzip content).

By the way the function of the no-gzip variable is also described directly on the mod_deflate page you linked to.

I agree.

Much better variable names than previous patches mistakenly introduced! (we still need to fix those)

Powered by Dreditor.

This looks good to me too. Could use some more eyes.

Am I right in assuming that the earlier rules are targeting already existing files and the following rules after are setting headers for not yet existing files?

Either way, seems like we need better comments in .htaccess then.

Minor: Could we move the first two lines above the IfModule? They seem to be the heading for the entire block.

Powered by Dreditor.

This one keeps the dropped L, and incorporates Sun's suggestion from #132.

Just to clarify, any reason why the Multiviews feature cannot work in that case?

#134: drupal-101227-134-css-js-gzip.patch queued for re-testing.

Another module I need to port to 7 now. I would really like it if core handled this; this patch is the more efficient and reliable solution.

Patch worked fine for me too.

And i don't see a reason to delay it until 8.x either. Most coredevs approved, the patch even got some polished documentation in the files and the results are outstanding ( at least on my shared host environment ).

+1 for drupal 7.0 beta 1 !

To compliment that, after applying this patch to my drupal 7 test site my co-devs commented in irc about that the site feels faster at the initial load ( and i didn't say a word about that patch! ). So thanks again for all patch suppliers and hope to see it in D7 stable.

Automatically closed -- issue fixed for 2 weeks with no activity.