Basic gzip support for 7x

I second this. I'm not sure why this is considered such a low priority... so low that it's marked in the 7.x handbook as "depreciated?"... when so many sites I see use gzip actively... but I find small file sizes really boost my site load speed. I know, shocking. Pardon that sarcasm... but depreciated?

Anyway, it's my understanding that checking for a header and enabling gzip doesn't take much code so I hope someone will add this feature in the near future. I'm going to tackle it myself on my site for my needs, but I don't know what's optimal for boost really (very new to it). So awesome module, can't wait for this "little" feature to come back.

I'm not sure why the handbook mentioned that (it may be my bad). Fixed it to link to this issue.

Can you please tell me in which particular file, you modified the code as above? thanks

Any patches possible?

Add it to the end of boost_exit(), AND, don't forget to modify your .htaccess file so that \.html in the 'gzip' section actually reads, \.html\.gz

At least, that worked for me.

subscribing

If you want this to move forward, please provide a patch, it saves us maintainers a lot of time in figuring out exactly the details.

It takes just a few minutes:

* clone the git repo:

git clone --recursive --branch 7.x-1.x http://git.drupal.org/project/boost.git
cd boost

* apply change to boost.module file

* git diff > /tmp/boost-gzip-1416214.patch

* attach patch file to this post.

Once we have a basic patch, we can move it forward from that point and solve the small details.

If you're working under Windows and installing git or using the command line is not obvious (although there are GUI solutions, but I don't know them much), just send a copy of your modified boost.module and we will do a diff/patch.

Thanks in advance :)

Inspired on the code from ralf I've created the attached patch.

It creates de gz version on cache folder and changes de htaccess generator.

Take a look please.

Thanks!

(using 7.x-1.x version)

After apply the patch, the boost admin area was gone. I just receive a HTTP Error 500.

Thanks

pmichelazzo that's probably a PHP error, please if you can give us more details it would help! :D

Committed to 7.x-1.x. Thanks for the patch!

Added the following:
* in admin/config/system/boost, added an option to disable gzip compression (enabled by default).
* compress only if the settings say so.

TODO:
* please test!
* test whether the cron will delete/expire the gz files as well.
* test 'expire' hook to see if it expires gz files.

Thanks for the quick feedback. I commented out temporarily the "gzencode" line in boost.module until we have more tests done.
Still requires however to clear the boost cache to delete the .gz files.

Were the rules in your main .htaccess (in the Drupal root directory) updated? (yep, I had forgotten to mention this rather important detail). -- we will need a way to remind people upgrading that their htaccess needs to be updated.

There's also the fact that cache is not expired by cron/expire, I presume, so gzip could show old content.

If it can help debugging, the most important rules for gzip are these:

  # GZIP
  RewriteCond %{HTTP:Accept-encoding} !gzip
  RewriteRule .* - [S=1]

  RewriteCond %{DOCUMENT_ROOT}/cache/%{ENV:boostpath}/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}\.html\.gz -s
  RewriteRule .* cache/%{ENV:boostpath}/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}\.html\.gz [L,T=text/html,E=no-gzip:1]

So the first part should read: "if client does not support gzip, skip the next rule".
The second part has "E=no-gzip:1", which disables mod_deflate to avoid double-compression.

Although, to be honest, I haven't dived into mod_rewrite in a while, and most of this was already implemented by mikeytown2, I only added the .gz suffix.

fwiw, I often use "wget" with the "-S" option (show response headers) to do tests and see headers:

wget -S -O /dev/null http://example.org/

Or to request a page with gzip enabled:

wget -S --header="Accept-Encoding: gzip" -O index.html.gz http://example.org/

(or rename the resulting page to foo.gz, so that it's then possible to open with "vim" or gunzip the file)

As per bgm suggestion on http://drupal.org/node/1888736#comment-6939904

I installed latest dev version of boost 7.x from 01/07/2013 date.

After uncommenting the line at 330, I could see .gzip files generated in cache/normal/... folder.

But when I tried fetching them in other browsers (FF & Safari) as anonymous user, I didn't see <---- boost comment line at the end indicating that page delivered is a cached paged --->

Secondly, how do I make sure that gzip version of file is delivered to http request?

I can do some more tests if you want.

Not at all silly because I missed it. :)

I kept .htaccess settings from my earlier boost installation.

I am using localhost. Do you think will it still matter to change .htaccess settings?

Hi,

I added .htaccess changes generated from default settings under boost.

Didn't have to change any RewriteRule.

I see .gz file generated in cache. Also tested it on pageSpeed. I observed 2 things:

1. In FF, if I enabler Cache, page speed rank is ~78/100. Under Resouces tab, it shows File size = 48k and Transfer size = 8k.

2. In FF, if I disable all Caches, page speed score increases to 98/100. Same values under Resource tab.

File size and Transfer size are for main html page. ( not including js, css, images etc).

Can anybody explain why it's giving better results without browser cache?

I didn't test it under Chrome.

But doing it on Safari now.

Will keep you all posted.

Similar functioning on Safari as well.

Clear cache under performance page, is cleaning all caches which is good.

I am still not sure if this is ready for production site yet.

Though I noticed an unusual behavior.

After boost caching turned ON and sign off , close the browser & then restart the browser.

Clear browser cookies, cache.

So now there's no cache generated yet in boost and browser is also clean.

Now, if start opening pages on site, it caches 2 pages only and rest of them are ignored irrespective of how many times you try, or even on different browser.
All this time, I was browsing as anonymous user but still no cache pages are created under boost dirs.

The only change I did apart from Boost is that I turned Blog comments On. But I haven't visited Blog content yet as anonymous user.

Using FF and Safari. Not sure if this is Boost related or something else.

Do not see DRUPAL_UID cookie set for anonymous user. As soon as I login, I see it and goes off when I log off. I don't think this is the root of above problem.

Besides, this I just installed HTTPRL & Cache Expire & enabled Boost Crawler.
As per instructions, on Cron run it should generate cache but it doesn't seem to be doing it.

I also tried changing a node and saving it, then running manual cron but no luck.

Is this a known issue?

You mean it will only crawl pages who's visible content is changed or we can just click edit and save without modifying actual content?

Does crawler checksum old and new file for differences?

Moreover normal boost isn't caching user/login & user/register pages. I tried many times but it's not.
Is it intended functionality or a bug?

Ah, I see, this totally makes sense. Thanks.

I faced an unusual issue today with Boost.

I have last recommended release installed with BOTCHA 7.14x

There are almost hundred files starting as "boostMxseY", "boostYgRkt", etc.

Any idea what could be causing it?

The problem with junk BoostZWRse files is resolved here http://drupal.org/node/1890836#comment-6957440.

It was an issue with Botcha module
Thanks.

I installed the latest Dev version, uncommented line 330 in boost.module and set the .htaccess rules. But when I test with Pagespeed Insights in Chrome it's still complaining about uncompressed javascripts and CSS files.

Am I missing something?

Cheers,
Bram.

p.s. You can see for yourself at http://intermin.bramdeleeuw.nl/

ah, appearantly I misunderstood. So I should comment-out line 330 again?

Thanks Philip for your info. I will get into it!

@Philip_Clarke: I've reproduced this issue, especially your finds from #14. It is correct that the current generated htaccess rules depend on mod_mime to be present in the apache installation.

When I'm not mistaken mod_mime is very important for most setups. I cannot think of any case where this module needs to be disabled, rather I think that an apache-configuration without this module will result in all sorts of trouble. IMO we could count on this module being enabled by default.

That said I suggest to just document the issue (i.e. serving compressed content with apache requires mod_mime). Also it would be possible to move the AddEncoding gzip .gz directive out of the IfModule condition in the cache .htaccess. That way users would get error messages when things are not configured properly instead of a gzip-data displayed in a browser window.

Another solution would be to do what core does and implement the same checks like core .htaccess based on mod_headers.

The drupal-core approach is not too difficult. I propose the following solution:

This is .htaccess in drupal root. Note that we simply set the variable boosttrygzip when both of the following conditions are met: the server has mod_headers and the client requests gzip.

  ### BOOST START ###

  RewriteRule .* - [E=boosttrygzip:0]
  <IfModule mod_headers.c>
    # Try to serve gzip compressed files if they exist and the client accepts
    # gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteRule .* - [E=boosttrygzip:1]
  </IfModule>

  # Allow for alt paths to be set via htaccess rules; allows for cached variants (future mobile support)
  RewriteRule .* - [E=boostpath:normal]

  # Caching for anonymous users
  # Skip boost IF not get request OR uri has wrong dir OR cookie is set OR request came from this server OR https request
  RewriteCond %{REQUEST_METHOD} !^(GET|HEAD)$ [OR]
  RewriteCond %{REQUEST_URI} (^/(admin|cache|misc|modules|sites|system|openid|themes|node/add|comment/reply))|(/(edit|user|user/(login|password|register))$) [OR]
  RewriteCond %{HTTPS} on [OR]
  RewriteCond %{HTTP_COOKIE} DRUPAL_UID [OR]
  RewriteCond %{ENV:REDIRECT_STATUS} 200
  RewriteRule .* - [S=3]

  # GZIP
  RewriteCond %{ENV:boosttrygzip} 0
  RewriteRule .* - [S=1]
  RewriteCond %{DOCUMENT_ROOT}/cache/%{ENV:boostpath}/localhost%{REQUEST_URI}_%{QUERY_STRING}\.html\.gz -s
  RewriteRule .* cache/%{ENV:boostpath}/localhost%{REQUEST_URI}_%{QUERY_STRING}\.html\.gz [L,T=text/html,E=no-gzip:1]

  # NORMAL
  RewriteCond %{DOCUMENT_ROOT}/cache/%{ENV:boostpath}/localhost%{REQUEST_URI}_%{QUERY_STRING}\.html -s
  RewriteRule .* cache/%{ENV:boostpath}/localhost%{REQUEST_URI}_%{QUERY_STRING}\.html [L,T=text/html]

  ### BOOST END ###

The cache-.htaccess then fixes content-encoding and vary headers when compressed content is to be delivered:

AddDefaultCharset utf-8
FileETag MTime Size
<FilesMatch "\.(html)(\.gz)?$">
  <IfModule mod_expires.c>
    ExpiresDefault A5
  </IfModule>
  <IfModule mod_headers.c>
    Header set Expires "Sun, 19 Nov 1978 05:00:00 GMT"
    Header unset Last-Modified
    Header append Vary Accept-Encoding
    Header set Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
    Header set X-Cached-By "Boost"
  </IfModule>
</FilesMatch>
<IfModule mod_mime.c>
  AddCharset utf-8 .html
</IfModule>
<FilesMatch "\.html(\.gz)?$">
  ForceType text/html
</FilesMatch>

<IfModule mod_headers.c>
  <FilesMatch "\.gz$">
    Header set Content-Encoding gzip
    Header append Vary Accept-Encoding
  </FilesMatch>
</IfModule>

SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
Options None
Options +FollowSymLinks

This setup works with any combination of mod_mime and mod_header. If the latter is missing, no gzip-content will be served though, but that's exactly what drupal core is doing anyway.

I'll have to test that later, but I assume that you've tested using chrome?

Yes.

Also how are you gzipping the files ? have you uncommented the section out from the last dev build.

Yes, patch attached.

@Philip_Clarke, thank you for the review.

The good news is that the original bug report I made about gzip not working with mod_headers disabled #1416214-14: Basic gzip support for 7x seems to have been solved in either chrome or apache 2.22 (although that does not mean that the bug would not reoccur in real life across many servers).

The reason that Chrome chokes on gzip content when mod_headers is disabled is not a bug in Chrome or Apache. If mod_headers is missing, Content-Encoding cannot be added to the response and the browser is not notified that it needs to decompress the data before attempting to render it. Drupal core .htaccess implements the exact same check when determining whether gzip-compressed CSS and JavaScript files should be served to the browser - i.e. check whether the browser accepts gzip and also check whether mod_headers is present. Only when both of these conditions are met, an attempt to serve compressed content (with appropriate headers) is made. If there were bugs in Chrome / Apache in this area, Drupal core would be affected too. I'm sure that the patch fixes the issue. IMO there is not much risk that this problem will crop up again on production systems when this fix gets in.

The only other modification was to remove Vary Encoding as it appears in the

<IfModule mod_headers.c>

code block above regardless as to whether zipped encoding is chosen or not.

Oh, good catch. It is actually very important that this header is added, regardless of whether the content is compressed or not.

I think the patch from #53 should get in.

Anyone else running into a situation where the first page loads fine, but any subsequent page loads (to anywhere on the site) results in the browser wanting download the gz file instead of rendering it? I've had the problem happen in FF, chrome and IE. Usually a drupal cache flush through admin menu fixes the problem, but it intermittently crops back up...

Bummer...

I'm showing both mod_mime and mod_headers to be compiled and loaded on Apache 2.2.24 :-/

Please note: After applying the patch you need to recreate all .htaccess files. The one in the drupal root as well as all those within the cache directory. Otherwise the patch will not have any effect.

The mechanism implemented in the patch is the same as the one Drupal 7 core uses for serving compressed js and css files, therefore it should work and break the same way as aggregated and compressed js/css files do.

Some of the early posts in this thread (up to about 15) detail that files sent out were being sent with incorrect lengths in the headers

It would help if you'd try to reproduce your finds from comment 14, then apply the patch, regenerate .htaccess files and retest.

In my situation, it wasn't specific to IE... I initially had the same problem using an incognito window in Chrome.

I updated the core .htaccess to accompany the patch, right after applying the patch. I'll look into the cache .htaccess for possible answers...

I'll look into the cache .htaccess for possible answers...

You need to make sure that the following fragment is in every .htaccess file in the cache directory:

<IfModule mod_headers.c>
  <FilesMatch "\.gz$">
    Header set Content-Encoding gzip
  </FilesMatch>
</IfModule>

@Philip_Clarke: If you're referring to that bit when you were saying that "The patch does try and force the file/ mimetype", then that's probably the source of confusion. This fragment is essential configuration when delivering any gzipped content intended to be unzipped by the browser. I initially had problems to understand the difference between Content-Type and Content-Encoding headers. The former refers to the payload (mime-type, character-encoding, etc) while the latter refers to the method used to compress the payload.

I do not see the ForceType directive in the patch. Therefore I do not understand how that is related to this issue.

Although I don't recall there ever having been a ForceType application/gzip

Such a configuration actually would be very wrong. ForceType affects the Content-Type header and not Content-Encoding.

In my opinion we do not need to make it work on configurations where core would fail to deliver compressed css/js files.

Added code example

Hi,
Currently I used Authcache with FileCache for Logged-in User whereas Boost for Anonymous users. I found that Authcache does cache for Anonymous users.

I check the cache file generated by Authcache with FileCache is about 10kb++ as compared to Boost which is around 40KB++, so far Boost is running perfectly for Anonymous users but what's bad it does not support compression.

Is it good to abandon Boost and go for Authcache with Filecache completely for the sake of page compression and better bandwidth management ? Sometimes, I'm really afraid of all these patches coz the site would encounter a serious breakdown after upgrade when the latest version does not include the patches made. Thanks

#69, thank you. Indeed, Boost is working lightning fast without issues now, I'm also very satisfied with it. However, I'm not so familiar with the technical issues as discussed above and thought that if Authcache would be a better replacement without tweaking and patching since it does support anonymous caching with compression. I don't really know much about the in-depth mechanisms being applied by Boost & Authcache, just configured and they work. Anyway, thanks for the comments.

The link from #69 is broken. Is there another resource? I think it would make scene to add this to the Handbook.

Before thinking of gzip:

1. Check that your server has mod_headers and mod_mime enabled.

2. Navigate to Boost module folder

3. File boost.module:

Uncomment lineboost_write_file($_boost['filename'] . '.gz', gzencode($data, 9));

Replace

if (unlink($filename)) {
  boost_log('Removed !file from the boost cache.', array('!file' => $filename), WATCHDOG_DEBUG);
}

with

if (BOOST_GZIP && file_exists($filename.'.gz')) {
  unlink($filename.'.gz');
}
if (unlink($filename)) {
  boost_log('Removed !file from the boost cache.', array('!file' => $filename), WATCHDOG_DEBUG);
}

Replace

elseif ($flush) {
  unlink($file);
  $counter++;
}

with

elseif ($flush) {
  unlink($file.'.gz');
  unlink($file);
  $counter++;
}

Replace

if (isset($lifetimes[$ext]) && $age > $lifetimes[$ext]) {
  unlink($file);
  $counter++;
}

with

if (isset($lifetimes[$ext]) && $age > $lifetimes[$ext]) {
  unlink($file.'.gz');
  unlink($file);
  $counter++;
}

4. File boost.blocks.inc:

Replace

if (unlink($filename)) {
  drupal_set_message(t('%filename was deleted from the Boost cache', array('%filename' => $filename)));
}

with

if (BOOST_GZIP && is_file($filename.'.gz')) {
  unlink($filename.'.gz');
}
if (unlink($filename)) {
  drupal_set_message(t('%filename was deleted from the Boost cache', array('%filename' => $filename)));
}

Done.

Might it be that #74 is causing the problems described in https://www.drupal.org/node/2421879 ?

69 and 75 not working for me. Is there a fix that is understandable anywhere??