Gzipping of static html files (*.html.gz)

Very interesting. I'll try to add this feature (optional) to 5.x branch.

Staffan, that's a nice idea and it would be a pretty useful feature, I think.

However, the problem with Boost is that it is already quite complicated to install and get working, for an average Drupal user. Any added complexity would increase the support burden further still, so I'm a bit cautious about adding new features - especially those related to Apache and .htaccess.

If somebody submits a well-thought-out patch (including documentation changes!) that implements gzip support, I will consider it. But I will not likely be implementing this myself.

Subscribing to this. I think this is the next logical step for this module to further "boost" the site. Obviously Arto's comments are valid, unless documentation is in place along with the code then it would make a great additional (optional) feature.

White-space removal would also be of great help on a busy website!

I hope this feature will be available soon, really makes your pages load fast!

I big +1 for this feature request.

I think I am the only user that has slower page loads using boost (although I am very happy with it, it is a precious module to deal with hosting providers :-))

I have a quite acceptable fine tuned site, and this feature gets critical when talking about page load times, to the point that boost loads pages slower than normal cache system just beacuse of lacking gzip compression (you know, you first load the page, and then the rest of components, so the sooner you get the main page, the faster the whole page loads).

I am not quite good at programming, but I think I am going to give it a try...

BTW, thank you Arto for this excelent module.

Here is a proposal for a very simple way to provide gzip compression.

It would be quite simple to enable compression changing the file extension from ".html" to, for instance ".html.gz". In that case, just adding one line to boost.api.inc file, we could have compressed files. To make the changes very, very easy, the package could be distributed with 2 versions of .htacces, one for standard ".html", and another one in case you want activate compression using ".html.gz" extensions.

The line to be added to boost.api.inc would be inside the funtion boost_cache_set. Instead of this:

  $alias = drupal_get_path_alias($path);
  $path = drupal_get_normal_path($path); // normalize path
  // Create or update the static file as needed

we would use this:

  $alias = drupal_get_path_alias($path);
  $path = drupal_get_normal_path($path); // normalize path

  if (BOOST_FILE_EXTENSION == '.html.gz') {$data = @gzencode($data, 8);} 

  // Create or update the static file as needed

Once this is done, and the files extension changed in the admin panel of the module, the .htaccess file should be modified to reflect ".html.gz" extensions, and to check if the requester supports gzip compression
ReWriteCond %{HTTP:accept-encoding} (gzip.*)
And finally, one last modification to add gzip Content-Encoding headers for .html.gz files
AddEncoding gzip .html.gz

I have done very little testing, but seems it works. I don't know the internals of the module so Arto, what do you think?

This solution has been working for almost 2 days in the production environment of Cuentos para Dormir, a spanish page with short children stories with about 2000 visits a day. It is not drupal.org, but it is quite acceptable test site.

The gzip compression (and the boost module) are working fine, and no problem has been reported up until now. And, you know, the page loads quite faster... ;-)

Pedropablo - where should i insert the line you mentioned into .htaccess ?

Anyone else using this with success?

This sounds interesting

Yes, it would be great to see what exactly needs to be changed in .htaccess It doesn't work for me for some reason - browser trying to save page instead. I think my server doesn't give it gzip headers.. Do I have to enable some Apache modules for this to work?

Thanks a lot!

Here is example of .htaccess for drupal6 which will work with pedropablo's patch

Great! Works just awesome! Thanks a lot!

Interesting fix; although I've noticed something odd about its behavior:

When I run boost without the compression patch (i.e. the change to boost.api.inc and the associated .htaccess file), it generates the cache files just fine; and when I reference a cached page, apache serves it up without any sort of noticable spike in CPU utilization.

However, when I apply the patch (by modding the .inc file, using the .htaccess file mentioned above and changing the file extension in admin->boost), I notice that whenever a page is referenced (even those that are cached and compressed), apache CPU utilization spikes to between 20 and 30% file the page is being served.

Any idea what's causing this behavior? It's acting like it's rebuilding and/or compressing the file when it shouldn't need to...

Thanks

figured out the issue...The cache directory structure used in the patch file is slightly different than the default used by boost. After adjusting a few of the RewriteCond and RewriteRule statements to account for this difference, everything worked fine.

@rcclarke, sorry, i forgot to say my htaccess is for boost6x. There are difference in cache directories between boost5 and boost6. So your pages were dynamically generated by drupal, not taken from cache.
For 5x you need to replace /cache/ with /cache/0/ in all RewriteCond and RewriteRule

If it's for the 6.x-1.x-dev version...

Another vote to have this added as an optional feature. I have to get Boost running first, but will then see what it takes to do.

Ideally this should create 2 files, one for gzip supported browsers (*.html.gz), one for non supported (*.html); just like Javascript Aggregator does #290280: Add GZip compression. Can you rewrite the already rewritten redirect? example.com is forwarded to example.com/cache/example.com/index.html, can you rewrite that to example.com/cache/example.com/index.html.gz? in short, add .gz at the end of the html file.

This is what I'm thinking of for a patch for 6.x

Modify boost_cache_set(). Use Drupal's file_save_data() instead of PHP's file_put_contents(). Mimic a lot of the logic found in javascript_aggregator_preprocess_page(). Each cached page gets 2 versions; normal (*.html) and gzipped (*.html.gz). Let apache serve either one based on RewriteCond %{HTTP:Accept-encoding} gzip, among some other rules. Include a "gzip boosted html files" checkbox on admin/settings/performance. Going to wait until the next dev because implementing this should happen after this (#174380: Remove symlink creation. Let each path have own file) is in the 6.x dev and has been tested.

Thoughts?

Since the /cache dir is excluded via RewriteCond, we can add a C to the end of boost's rewrite rule (see chain http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewriterule). Then the final boost rewrite rule is something like

#serve gziped content if it exists
<FilesMatch "\.(html.gz)$">
  AddEncoding x-gzip .gz
  ForceType text/html
</FilesMatch>
RewriteCond %{REQUEST_URI} ^/cache/%{SERVER_NAME}
RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
RewriteCond %{HTTP:Accept-encoding} gzip
RewriteCond %{REQUEST_FILENAME}.gz -f
RewriteRule ^(.*)\.html $1.html.gz [L]

#finally serve file if it exists inside the cache dir
RewriteCond %{REQUEST_URI} ^/cache/%{SERVER_NAME}
RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule .? - [L]

Thoughts?
skip is another way, that might be better then chain.

Just tested and skip is the correct way to get this done. Above rules when added to the bottom of the boost part allow for normal operation when .gz files are not present. Next step is to create the gzipped html along with the normal html.

Here are my rewrite rules

  # BOOST START
  <IfModule mod_headers.c>
    Header add Expires "Sun, 19 Nov 1978 05:00:00 GMT"
    Header add Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0"
  </IfModule>
  <IfModule mod_mime.c>
    AddCharset utf-8 .html
  </IfModule>
  # root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/$
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}/index.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/index.html [S=3]
  # subdir root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI} -d
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}/index.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}/index.html [S=2]
  # non root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}.html [S=1]
  # url variables
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} !^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}_%{QUERY_STRING}.html [S=0]
  
  #serve gziped content if it exists
  <FilesMatch "\.(html.gz)$">
    AddEncoding x-gzip .gz
    ForceType text/html
  </FilesMatch>
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}.gz -f
  RewriteRule ^(.*)\.html $1.html.gz [L]
  # BOOST END

Thinking about this and this will be double trouble for people that have lots of nodes in a folder already. would having a separate gz dir get us back to where we were? #410730: System limits: Number of files in a single directory

cache/example.com/taxonomy/index.html
maps to
cache/example.com/taxonomy/gz/index.html.gz
or
cache/example.com/gz/taxonomy/index.html.gz
or
cache/gz/example.com/taxonomy/index.html.gz

cache/gzip/example.com/taxonomy/index.html.gz is the right way to do this I think... means we need 8 rules instead of the above 5. Should also be a setting on the performance page to click on/off gzip functionality.

Split rules up into 8+1. 1st rule is to serve the file if it exists on the server. It's a performance thing from #276495: Update for Rewrite Rules - Give Boost even bigger performance gain that's safe. Next step is changing the php code so the .gz files get written to their own dir; and make sure that dir gets cleared.

  # BOOST START
  <IfModule mod_headers.c>
    Header add Expires "Sun, 19 Nov 1978 05:00:00 GMT"
    Header add Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0"
  </IfModule>
  <IfModule mod_mime.c>
    AddCharset utf-8 .html
  </IfModule>
  <FilesMatch "\.(html.gz)$">
    AddEncoding x-gzip .gz
    ForceType text/html
  </FilesMatch>
  #serve file IF it exist on server
  RewriteCond %{REQUEST_FILENAME} -f
  RewriteRule .? - [L]
  
  #GZIP
  # root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/$
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}/index.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/index.html.gz [L]
  # subdir root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI} -d
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}/index.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}/index.html.gz [L]
  # non root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}.html.gz [L]
  # url variables
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} !^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}_%{QUERY_STRING}.html.gz [L]
  
  #NORMAL
  # root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/$
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}/index.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/index.html [L]
  # subdir root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI} -d
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}/index.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}/index.html [L]
  # non root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}.html [L]
  # url variables
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login
  RewriteCond %{QUERY_STRING} !^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}_%{QUERY_STRING}.html [L]
  # BOOST END

There where some dumb errors in the above code... should be fixed now
RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
vs
RewriteCond %{REQUEST_URI} ^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login

  # BOOST START
  <IfModule mod_headers.c>
    Header add Expires "Sun, 19 Nov 1978 05:00:00 GMT"
    Header add Cache-Control "store, no-cache, must-revalidate, post-check=0, pre-check=0"
  </IfModule>
  <IfModule mod_mime.c>
    AddCharset utf-8 .html
  </IfModule>
  <FilesMatch "\.(html.gz)$">
    AddEncoding x-gzip .gz
    ForceType text/html
  </FilesMatch>
  #serve file IF it exist on server
  RewriteCond %{REQUEST_FILENAME} -f
  RewriteRule .? - [L]
  
  #GZIP
  # root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/$
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}/index.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/index.html.gz [L]
  # subdir root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI} -d
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}/index.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}/index.html.gz [L]
  # non root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}.html.gz [L]
  # url variables
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
  RewriteCond %{QUERY_STRING} !^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP_USER_AGENT} !".*Safari.*"
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html.gz -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}_%{QUERY_STRING}.html.gz [L]
  
  #NORMAL
  # root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} ^/$
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}/index.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/index.html [L]
  # subdir root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI} -d
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}/index.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}/index.html [L]
  # non root
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
  RewriteCond %{QUERY_STRING} ^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}.html [L]
  # url variables
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{REQUEST_URI} !(^/admin|^/cache|^/misc|^/modules|^/sites|^/system|^/themes|^/user/login)
  RewriteCond %{QUERY_STRING} !^$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html -f
  RewriteRule ^(.*)$ cache/%{SERVER_NAME}/%{REQUEST_URI}_%{QUERY_STRING}.html [L]
  # BOOST END

Here's a patch that contains a gzip enable/disable setting and it moves the html.gz files to the cache/gz dir. htaccess Rules are in boosted1.txt/boosted2.txt. Please Test!

committed