Gzip aggregated CSS and JS

Owen Barton - December 4, 2006 - 10:03
Project:Drupal
Version:7.x-dev
Component:base system
Category:feature request
Priority:normal
Assigned:Unassigned
Status:needs review
Issue tags:Performance
Description

This is a follow on from #100516 - CSS preprocessor (and, originally #81835), which is a patch to aggregate multiple CSS files into a single (cached) file.

This patch (which should be applied on top of the #100516 patch):

  • Adds an option to the settings to gzip this cached file.
  • If enabled then it gzips and saves a .css.gz file in addition to the regular .css file and...
  • Adds a .htaccess rule to use this file if the browser accepts gzip and the gz file exists.

As you can see, it is a very simple addition, but should be very valuable for heavy sites. Apart from the time saved for users, there is a 16Kb bandwidth saving too - potentially several GB of bandwidth a year for a site with several 100K visitors.

I currently have this flagged as a feature request for 5.0, but it could very well be considered a usability enhancement.

Here are my original benchmarks:
-----------------------------------------

<?php
          HEAD
          Total       Transfer Duration       Page Duration
Test 1    12798       8756                    8454
Test 2    12984       8907                    8162
Test 3    12730       8718                    7966
Average   12837       8794                    8194
Baseline  100
%        100%                    100%
                    
         
conditional_css_include_2.patch
          Total       Transfer Duration       Page Duration
Test 1    10633       7111                    6828
Test 2    11225       7332                    7530
Test 3    11047       7639                    7357
Average   10968.33    7361                    7238
Faster By 15
%         16%                     12%
                    
         
cache_19.patch
          Total       Transfer Duration       Page Duration
Test 1    9160        5453                    5175
Test 2    9417        5712                    4961
Test 3    9056        5306                    5290
Average   9211        5490                    5142
Faster By 28
%         38%                     37%

         
conditional_css_include_2.patch AND cache_19.patch
          Total       Transfer Duration       Page Duration
Test 1    9595        5615                    5333
Test 2    9909        5709                    5425
Test 3    9461        5824                    5537
Average   9655        5716                    5432
Faster By 25
%         35%                     34%

         
cache_19.patch AND gzip
          Total       Transfer Duration       Page Duration
Test 1    7429        3495                    2741
Test 2    7027        3306                    2758
Test 3    6915        3267                    3001
Average   7124        3356                    2834
Faster By 45
%         62%                     65%

         
conditional_css_include_2.patch AND cache_19.patch AND gzip
          Total       Transfer Duration       Page Duration
Test 1    7489        3157                    2395
Test 2    6699        3107                    2364
Test 3    7250        3013                    2255
Average   7146        3092                    2338
Faster By 44
%         65%                     71%
?>

* Testing was done locally, to eliminate the variable latency you get on live networks. The Charles web debugging proxy was used to apply a consistent throttle and latency equivalent to a typical 64Kbps connection.
* Browser caching was disabled completely, to simulate an initial page load. I can repeat, with caching enabled (to test http cache freshness checks) if that would be useful.
* Drupal page caching (css caching for #100516) was enabled, and the test was done as an anonymous user. All Drupal modules were enabled (a few of these would more likely be contrib modules in reality). The page cache was cleared before each set of tests, and 2 dummy reloads were made before starting timing.
* The first column 'Total' is the total time spent transferring data, as reported by Charles. This does not include the time saved by browser pipelining of http requests.
* The second column 'Transfer Duration' is the Duration between the first byte transfer and the last byte transfer, as reported by Charles
* The third column 'Page Duration' is the time between the end user hitting refresh and Firefox finishing building the page. This is occasionally less that the transfer duration, which is a little odd, but perhaps certain page graphics (favicons maybe?) are not included in the page build time.

Note that these percentages are compared to the times in my original HEAD benchmark (with no patches) at http://drupal.org/node/100516#comment-162025
Please read the notes on that comment if you have not done so already. Also note that all the conditional css includes patch is doing here is somewhat reducing the cached css size, because none of the conditions are met on the front page.

Here is a summary:

<?php
Test Set                     Faster By
Seconds Saved:
HEAD                         100%        0
conditional                  12
%         1
caching                      38
%         3
conditional
+ caching        34%         3
caching
+ gzip               65%         5.3
conditional
+ caching + gzip 71%         5.8
?>

Looking at this I am actually wondering if we should be looking at including gzipped css (and js) in core - if not for 5.0 - then certainly in 6.0. While the percentages are skewed by the fact that my test page is pretty lightweight (i.e. not much content, and no user added images) the seconds saved are real, actual seconds that would be saved by a user on a 64Kbps connection, and would be saved no matter what is added to the site and theme in the way of content and additional images. Also note that the conditional includes patch has now been committed to HEAD, but wasn't when I did my initial benchmarks - I am keeping it separate here for easier comparison.

Bearing in mind the statistic that most users will only wait 4 seconds before going to a different site, the application of aggregation/caching and gzip can take the initial page load time from a very poor 7 or 8 seconds, to a very respectful 2 seconds. The difference between these patches is extremely noticeable, the site goes from feeling pretty sluggish to appearing extremely fast.

AttachmentSize
gzip_css_1.patch.txt2.62 KB

#1

lennart - December 4, 2006 - 12:13

Personally, I think this is critical. Initial page load time is a critical factor for first time visitors. This one almost cuts the initial load time in half. Very significant!

#2

Owen Barton - December 4, 2006 - 15:31

I just noticed a typo in the benchmark tables - 'Faster By' for HEAD should of course be 0% (or N/A!), not 100%.

#3

moshe weitzman - December 4, 2006 - 15:38
Status:needs review» needs work

we have outstanding bugs on our gzipped page cache (if first person to view a page has gzip disabled, we still cache that page and send wrong headers). i'd rather see that resolved before we add more gzip.

the implementation here is a bit more interesting since the logic is in .htaccess. is there no performance penalty for this .htaccess check on every request?

also, don't we have to tell the browser that we are sending gzip in the response headers? are we expecting browser to know because filename ends in .gz?

the #description doesn't explain anything about gzip

#4

m3avrck - May 24, 2007 - 02:53
Version:5.x-dev» 6.x-dev

Subscribing... this should handle CSS and JS files...

#5

Owen Barton - May 24, 2007 - 18:32

There is a different (better, I think) approach in my sandbox: http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/grugnog/gzip_... - which should deal with css, js and a few other things too.

I think moshe was right about the headers (in terms of following specs), so the .htaccess option is probably out (although it did work cross-browser).

#6

m3avrck - June 1, 2007 - 16:30

With 4 weeks to go, let's see if we can get this in :-)

Owen, did you want to start with a patch based off of your module? If not, I can try and post one in the next few days.

#7

catch - November 7, 2007 - 15:02
Version:6.x-dev» 7.x-dev

#8

Wim Leers - January 27, 2008 - 18:35

Subscribing.

#9

Wim Leers - January 27, 2008 - 18:48
Title:Gzip aggregated CSS to cut 2 seconds off initial load» Gzip aggregated CSS and JS

Since JS aggregation is now in Drupal 6 core as well, I suggest we tackle both CSS and JS aggregation here.

#10

satynos - February 18, 2008 - 05:07

subscribing...

#11

DanielTheViking - April 20, 2008 - 11:46

Subscribing.

#12

BioALIEN - April 21, 2008 - 02:47

This had huge potential but unfortunately didn't make it into 6.x. What's holding up the progress on this patch? Is it because of the outstanding bugs on gzipped page cache in core?

Also I agree with Grugnog2 in #5 about the use of .htaccess method, while it is browser compatible, Dries highlighted his desire to make core compatible with lighttpd and other web servers. We should put some consideration for those too if possible.

#13

oNyx - May 6, 2008 - 06:25

On my site a typical page is around 100kb in size. About 75kb of that are aggregated js (50kb) and css (25kb) files, which can be compressed down to about 25kb (with stripped comments 17kb).

Even without stripping comments it would cut it in half for first time visitors. And you can get a whole torrent of those from social bookmarking sites. My puny little page gets about 3000 of those occasionally and I'm currently on some cheap shared host. So, it generates about 300mb instead of 150mb. That's a pretty big difference if you ask me.

Since the number of visitors is constantly rising I really would love to see that kind of thing in core ASAP. Otherwise I'll need to switch to some more expensive hosting plan a few months sooner than actually necessary.

#14

alanburke - June 29, 2008 - 16:24

subscribing

#15

chirale - June 29, 2008 - 16:46

This topic is pretty hot nowadays, I read two related discussions on contrib modules for 5.x: this one minify the Javascript aggregated file, and this is about "statical" gzipping method. Today I start to use both the approaches combined, and the result is fine. Since aggregation is a core task, I leave this to a small bash script run on cron job. Should JSMin patch to Javascript Aggregator be reused here?

#16

oNyx - July 4, 2008 - 22:29

Got tired of waiting and just added it to my installation. Only requires 14 extra lines in .htaccess and 2 in common.inc. These 2 lines need to be added again after updating, but imo it's totally worth the hassle.

The step by step guide is over here:
How to GZip Drupal 6.x's aggregated CSS and JS files

#17

Susurrus - July 5, 2008 - 03:34

It seems that Zlib support isn't standard for a PHP installation, at least not on non-Windows OSes. Can someone confirm this?

If this is the case, it might stand in the way of adding this.

#18

oNyx - July 5, 2008 - 15:44

Surprisingly you're correct. It's silly, but there are indeed a few installations without Zlib support.

If it's implemented as an option, it would be a good idea to guard the setting (defaults to off) with a function_exists('gzencode') condition.

Fortunately the level parameter for gzencode exists since 4.2 and Drupal requires 4.3.5+. So, at least that part won't be an issue.

#19

Susurrus - July 5, 2008 - 16:48

Well, Drupal 7 requires 5.2+, so that's definitely not an issue. I would think this would need to be implemented like Clean URLs, where the option to enable gzip encoding would be disabled if the function doesn't exist with an explanation of why.

How about a reroll as an option in the Performance settings page?

#20

lilou - August 31, 2008 - 03:57

subscribe.

#21

AltaVida - November 4, 2008 - 04:42

subscribe

#22

Owen Barton - December 26, 2008 - 23:19
Status:needs work» needs review

Here is a patch, based somewhat on the original patch I posted, but with improvements based on http://kaioa.com/node/78 and http://drupal.org/node/290280

Here is how this should work
* It should fail safe to the regular uncompressed aggregated CSS or JS in almost every situation. This includes gzencode not being available (e.g. on very cut down windows PHP builds), if mod_rewrite is not available and enabled, if the gz file is not found or the browser does not accept gzip encoded data.
* It sends both the correct file extension (i.e. we don't ask the browser to request a .gz file, but just rewrite the request behind the scenes) and also the correct MIME type for the file.
* It does not involve any kind of PHP bootstrap at all - everything is done at the Apache level. In addition, the .htaccess files are created dynamically within the files/X directories, meaning that we don't invoke additional rewrite rule tests (a concern of Moshe in #3) or clutter up an already complex /.htaccess.
* There is no additional user interface cruft. We don't have options for core page level gzipping and I don't see any need for them here, as long as we give it sufficient testing cross-platform. This is a no-brainer.

I have tested this on my system and it worked very well, on the first run in fact (the general approach is also quite battle hardened on many production sites).

If you want to test this out you can do this by applying the patch and then checking the response headers using the "Net" tab in Firebug or a similar tool for both .css and .js files. You should see "Content-Encoding gzip" in there and also notice a noticeable drop in file size.

If someone would like to write some tests to polish this off that would be a valuable contribution.

AttachmentSize
gzip.patch 3.12 KB
Testbed results
gzip.patchpassedPassed: 11576 passes, 0 fails, 0 exceptions Detailed results

#23

Rob Loach - December 27, 2008 - 02:15
Status:needs review» needs work

Is there any reason why we're checking function_exists with gzencode? Drupal 7 requires at least PHP 5.2, so wouldn't it be safe to assume it's available?

As for tests, I'd love those to go in with this patch, but won't have the chance to hit it up until next week because I'll be off on vacation. Otherwise, great work! Also, does that clear cache functionality remove the respective .htaccess file?

#24

Owen Barton - December 27, 2008 - 14:38

@Rob Loach

We are testing function_exists with gzencode for the same reason we do this with the core page gzipping code. Zlib support for PHP is extremely common, but is actually not enabled in a default vanilla compile of PHP (even in 5.2), so it is quite possible that it will not be available - "Zlib support in PHP is not enabled by default. You will need to configure PHP --with-zlib[=DIR]" from http://www.php.net/manual/en/zlib.installation.php

If you have a chance to do some tests that would be awesome!

The cache clearing functions do not remove the .htaccess file - all dot files are excluded. This shouldn't matter either way, since the file would be recreated anyway and is harmless if no .gz files are present. Possibly we might want to add a line to delete the .htaccess file at the point when aggregation is disabled (although it causes no issue if it is there), or perhaps even the js/css directory in it's entirety (on the basis that it is cruft and could be confusing to a newbie).

#25

catch - December 27, 2008 - 22:45

We do have a UI for core page gzipping - see 'page compression' at admin/settings/performance - and it's a useful thing to have if you're using mod_deflate/mod_gzip etc. We could reword the interface text there and use the same variable for both things though - seems like it'd be the same reason for switching one off as switching off the other.

#26

janusman - January 6, 2009 - 15:09

subscribing

#27

kenorb - January 14, 2009 - 02:12

I'm not quite understand, why Drupal don't gziping all other pages, even browser have support for gip (I'm not talking about js and css).
I've tried to found it, but without result. Somebody know some link to the topic?

#28

Rob Loach - January 14, 2009 - 02:45

I think this should be solely be managed through contrib once #352951: Make JS Preprocessing Pluggable is in.

#29

btully - February 5, 2009 - 16:20

subscribe

#30

mikeytown2 - March 31, 2009 - 20:43

Update: Contributed modules can now do this for 6.x. Once the code freeze is in place (early September), porting these to 7.x shouldn't be that hard.
http://drupal.org/project/javascript_aggregator
http://drupal.org/project/css_gzip

#31

andypost - May 17, 2009 - 02:04

Both modules exclude safari (webkit) so first projects from #30 should be fixed

Safari (webkit) bug https://bugs.webkit.org/show_bug.cgi?id=9521

So there 2 solutions:

1) make filename.css with filename.css.gz.css

2) alternative way is store filename.css (gzipped) and plain filename.nogzip.css

Both work with safari 3 and google chrome

<IfModule mod_rewrite.c>
    RewriteEngine On
# Konqueror and "old" browsers
    RewriteCond %{HTTP:Accept-encoding} !gzip [OR]
    RewriteCond %{HTTP_USER_AGENT} Konqueror
    RewriteRule ^(.*)\.(css|js)$ $1.nogzip.$2 [QSA,L]
</IfModule>    

<IfModule mod_headers.c>
    Header append Vary User-Agent
# for all  css/js files setup Content-Encoding
    <FilesMatch .*\.(js|css)$>
Header set Content-Encoding: gzip
Header set Cache-control: private
    </FilesMatch>
# reset Content-Encoding if not archive
    <FilesMatch .*\.nogzip\.(js|css)$>
Header unset Content-Encoding
    </FilesMatch>
</IfModule>

#32

lilou - May 17, 2009 - 02:38

Add tag.

#33

mikeytown2 - May 17, 2009 - 19:19

#34

andypost - May 17, 2009 - 20:10

#35

mikeytown2 - May 25, 2009 - 22:01
Issue tags:-needs backport to D6

Been thinking about the safari/webkit bug... because of the rewrite rules, the browser only sees it as *.css where on the file system it is *.css.gz. Changing the extension around would have no effect; in short it's not a bug because the html file doesn't reference *.css.gz it still points to *.css (bug report deals with *.css.gz in the html code). With that in mind this is a fairly straight forward patch for D7, the question is will it be accepted if I write one?

There are 3 functional changes that need to take place
http://api.drupal.org/api/function/drupal_build_css_cache/7 - write *.css.gz files
http://api.drupal.org/api/function/drupal_build_js_cache/7 - write *.js.gz files
http://api.drupal.org/api/function/system_performance_settings/7 - add gzip compression settings

Add this to .htaccess, inside <IfModule mod_rewrite.c> right above # Rewrite URLs of the form 'x' to the form 'index.php?q=x'.

  <FilesMatch "\.(css.gz)$">
    AddEncoding x-gzip .gz
    ForceType text/css
  </FilesMatch>
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}.gz -f
  RewriteRule ^(.*)\.css $1.css.gz [L,QSA]
 
  <FilesMatch "\.(js.gz)$">
    AddEncoding x-gzip .gz
    ForceType text/javascript
  </FilesMatch>
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}.gz -f
  RewriteRule ^(.*)\.js $1.js.gz [L,QSA]

#36

chx - May 25, 2009 - 22:01

khm.

AddOutputFilterByType DEFLATE text/css application/x-javascript text/html text/plain text/xml

#37

mikeytown2 - May 25, 2009 - 22:31

@chx, that compresses on the fly for every request; slightly slower. Don't forget about level 9; default is 6 if I remember correctly.

AddOutputFilterByType DEFLATE text/css application/x-javascript text/html text/plain text/xml
DeflateCompressionLevel 9

Core caches gzip html, so it's not a way out there request; plus having settings in the UI is a nice usability feature. It's something most people want IMHO. Great point BTW.

#38

hass - June 3, 2009 - 21:17

#35 doesn't work in IIS.

#39

mikeytown2 - June 3, 2009 - 21:26

how does IIS support conditional gzip, so only the clients that support gzip get gzipped content?

#40

andypost - June 3, 2009 - 21:57

@hass IIS rewrite rules are possible different

#41

hass - June 3, 2009 - 22:29

This is not possible in IIS (for e.g. with Helicon ISAPI_Rewrite, and not at all without payed plugins) and therefore no general solution:

<FilesMatch "\.(css.gz)$">
    AddEncoding x-gzip .gz
    ForceType text/css
</FilesMatch>

#42

hass - June 3, 2009 - 22:27

@mikeytown2: IIS does not have any GZIP compression feature build in for dynamic files (only for static HTML files). You need to buy extra plugins to archive this - if you think you really need it...

#43

mikeytown2 - June 11, 2009 - 08:15

Shrunk gzip .htaccess rules. Escaped all periods . for possible performance increase with pattern matching since . means any character. Anyone wants to test the escaped period hypothesis out?

Option 1

  <FilesMatch "\.gz$">
    AddEncoding x-gzip \.gz
  </FilesMatch>
  #skip gzipped css/js files if browser doesn't accept gzip encoding
  RewriteCond %{HTTP:Accept-encoding} !gzip
  RewriteRule .* - [S=2]
  #CSS
  RewriteCond %{REQUEST_FILENAME}\.gz -s
  RewriteRule (.*)\.css$ $1\.css\.gz [L,QSA,T=text/css]
  #JS
  RewriteCond %{REQUEST_FILENAME}\.gz -s
  RewriteRule (.*)\.js$ $1\.js\.gz [L,QSA,T=text/javascript]

Option 2

  <FilesMatch "\.gz$">
    AddEncoding x-gzip \.gz
  </FilesMatch>
  #CSS
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}\.gz -s
  RewriteRule (.*)\.css$ $1\.css\.gz [L,QSA,T=text/css]
  #JS
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}\.gz -s
  RewriteRule (.*)\.js$ $1\.js\.gz [L,QSA,T=text/javascript]

Option 3 (no escaped periods)

  <FilesMatch "\.gz$">
    AddEncoding x-gzip .gz
  </FilesMatch>
  #skip gzipped css/js files if browser doesn't accept gzip encoding
  RewriteCond %{HTTP:Accept-encoding} !gzip
  RewriteRule .* - [S=2]
  #CSS
  RewriteCond %{REQUEST_FILENAME}.gz -s
  RewriteRule (.*)\.css$ $1.css.gz [L,QSA,T=text/css]
  #JS
  RewriteCond %{REQUEST_FILENAME}.gz -s
  RewriteRule (.*)\.js$ $1.js.gz [L,QSA,T=text/javascript]

Which way would be preferred: 1, 2 or 3?

#44

mikeytown2 - June 21, 2009 - 03:27
Status:needs work» needs review

Gzip setting controlled by page_compression. Trying to make this as simple as possible.

AttachmentSize
gzip-101227.patch 2.69 KB
Testbed results
gzip-101227.patchfailedFailed: Failed to install HEAD. Detailed results

#45

hass - June 21, 2009 - 11:13

What will happen if the server do not support GZ compression or any of this Apache rewrite rules?

#46

Owen Barton - June 21, 2009 - 14:50

The attached patch includes the same conditional checks as page gzipping, so should fail where php has no zlib. The .htaccess will only trigger for .gz files exist, so nothing should break there.

The only thing I was wondering is if the rewrite rules for this should also be within a block (which could also contain the mime rule) - because if rewrite is enabled but mime isn't it seems there is a possibility of the type being incorrect, right?

AttachmentSize
gzip-101227-2.patch 3.99 KB
Testbed results
gzip-101227-2.patchfailedFailed: Failed to install HEAD. Detailed results

#47

hass - June 21, 2009 - 15:48

This fails on IIS:

+<FilesMatch "\.js\.gz$">
+  ForceType text/javascript
+</FilesMatch>

But this will work:

+  # Serve gzip compressed js files
+  RewriteCond %{HTTP:Accept-encoding} gzip
+  RewriteCond %{REQUEST_FILENAME}\.gz -s
+  RewriteRule (.*)\.js$ $1\.js\.gz [L,QSA,T=text/javascript]

In such a case the ForceType may not send to the browser, but the compressed file is... I'm not sure if this could cause issues!?

#48

mikeytown2 - June 21, 2009 - 17:59

Due to apache weirdness I'm setting the type twice. Once globally in the FilesMatch argument; the other time with T=text/javascript in the RewriteRule. That seems to work 100% of the time on various versions of apache. Does IIS work with T= ?

#49

hass - June 21, 2009 - 20:22

#50

mikeytown2 - June 21, 2009 - 23:59

hass, good to hear that works. Here's a slight adjustment to #46, adjusted the regular expression for the rewrite rule so it compares from the start, instead of from the end. Should always work with *.css?5 and things like that. The reason I need both T=MIME-type and the FilesMatch argument is because the newest version of apache doesn't work with T=MIME-type; well you can get it to work if you do some ugly tricks, so FilesMatch is the best way to get around the apache inconsistencies.
http://httpd.apache.org/docs/trunk/mod/mod_rewrite.html#rewriterule

'type|T=MIME-type' (force MIME type)
Force the MIME-type of the target file to be MIME-type. This can be used to set up the content-type based on some conditions. If used in per-directory context, use only - (dash) as the substitution, otherwise the MIME-type set with this flag is lost due to an internal re-processing.

http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewriterule

'type|T=MIME-type' (force MIME type)
Force the MIME-type of the target file to be MIME-type. This can be used to set up the content-type based on some conditions. For example, the following snippet allows .php files to be displayed by mod_php if they are called with the .phps extension:

AttachmentSize
gzip-101227-3.patch 4.13 KB
Testbed results
gzip-101227-3.patchpassedPassed: 11576 passes, 0 fails, 0 exceptions Detailed results

#51

System Message - June 24, 2009 - 01:10
Status:needs review» needs work

The last submitted patch failed testing.

#52

mikeytown2 - June 24, 2009 - 07:38
Status:needs work» needs review

Another reason why this should be in core
http://code.google.com/speed/articles/gzip.html

Above patch should still be good... re-submitting it.

#53

System Message - June 27, 2009 - 20:10
Status:needs review» needs work

The last submitted patch failed testing.

#54

andypost - July 1, 2009 - 11:50
Status:needs work» needs review

Suppose bot was broken, so re-test

 
 

Drupal is a registered trademark of Dries Buytaert.