CSS & JS files not served from cache (long, sorry)

You are much better at "succinct" than I! yes that is the problem entirely.

I run about a dozen sites and growing and yes, it shows up for all the sites, even the main site using "sites/default/settings.php". Pretty easy to test, I just delete something like /sites/default/files/css/css_83175784505a60a68cd2f3d58d82d04e.css and refresh an anonymous view requiring that file ... and then look at the ugly results of no CSS at all. Same for js.

Considering that this must be a mod_rewrite problem, I'll let you know that the web sites are behind NATed firewalls sharing IPs ... clearly HTTP_HOST is being set as that is how the sites are being accessed and the rules you provide seem pretty straight-forward ... they ought to be working. Just to be clear, the cached html files ARE being delivered as expected.

I'm stymied.

It seems like this isn't too dangerous, but 20+ years worrying about computer security makes me nervous :-)...nonetheless, here you go

#
# Apache/PHP/Drupal settings:
#

# Protect files and directories from prying eyes.
<FilesMatch "\.(engine|inc|info|install|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl|svn-base)$|^(code-style\.pl|Entries.*|Repository|Root|Tag|Template|all-wcprops|entries|format)$">
  Order allow,deny
</FilesMatch>

# Don't show directory listings for URLs which map to a directory.
Options -Indexes

# Follow symbolic links in this directory.
Options +FollowSymLinks

# Make Drupal handle any 404 errors.
ErrorDocument 404 /index.php

# Force simple error message for requests for non-existent favicon.ico.
<Files favicon.ico>
  # There is no end quote below, for compatibility with Apache 1.3.
  ErrorDocument 404 "The requested file favicon.ico was not found.
</Files>

# Set the default handler.
DirectoryIndex index.php

# Override PHP settings. More in sites/default/settings.php
# but the following cannot be changed at runtime.

# PHP 4, Apache 1.
<IfModule mod_php4.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_value mbstring.encoding_translation   0
</IfModule>

# PHP 4, Apache 2.
<IfModule sapi_apache2.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_value mbstring.encoding_translation   0
</IfModule>

# PHP 5, Apache 1 and 2.
<IfModule mod_php5.c>
  php_value magic_quotes_gpc                0
  php_value register_globals                0
  php_value session.auto_start              0
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_value mbstring.encoding_translation   0
</IfModule>

# Requires mod_expires to be enabled.
<IfModule mod_expires.c>
#  # Enable expirations.
  ExpiresActive On

  # Cache all files for 2 weeks after access (A).
  ExpiresDefault A1209600

  <FilesMatch \.php$>
    # Do not allow PHP scripts to be cached unless they explicitly send cache
    # headers themselves. Otherwise all scripts would have to overwrite the
    # headers set by mod_expires if they want another caching behavior. This may
    # fail if an error occurs early in the bootstrap process, and it may cause
    # problems if a non-Drupal PHP file is installed in a subdirectory.
    ExpiresActive Off
  </FilesMatch>
</IfModule>
 Various rewrite rules.
<IfModule mod_rewrite.c>
  RewriteEngine on

  ReWriteCond %{HTTP_HOST} ^gatevillage\.net$ [NC]
  ReWriteRule ^(.*)$ http://www.gatevillage.net/$1 [L,R=301]

  ReWriteCond %[HTTP_HOST] ^.*gatevillage\.com$ [NC]
  RewriteRule ^(.*)$ http://www.gatevillage.net/$1 [L,R=301]

#  RewriteBase /
  ### BOOST START ###
  AddDefaultCharset utf-8
   <FilesMatch "(\.html|\.json)$">
     <IfModule mod_headers.c>
       Header set Expires "Sun, 19 Nov 1978 05:00:00 GMT"
       Header set Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
     </IfModule>
   </FilesMatch>
  FileETag MTime Size
  <IfModule mod_mime.c>
    AddCharset utf-8 .html
    AddCharset utf-8 .css
    AddCharset utf-8 .js
    AddCharset utf-8 .json
  </IfModule>

  # NORMAL - Cached css & js files
  RewriteCond %{DOCUMENT_ROOT}/cache/%{HTTP_HOST}%{REQUEST_URI}_\.css -s
  RewriteRule .* cache/%{HTTP_HOST}%{REQUEST_URI}_\.css [L,QSA,T=text/css]
  RewriteCond %{DOCUMENT_ROOT}/cache/%{HTTP_HOST}%{REQUEST_URI}_\.js -s
  RewriteRule .* cache/%{HTTP_HOST}%{REQUEST_URI}_\.js [L,QSA,T=text/javascript]

  # Caching for anonymous users
  # Skip boost IF not get request OR uri has wrong dir OR cookie is set OR request came from this server 
  RewriteCond %{REQUEST_METHOD} !^GET$ [OR]
  RewriteCond %{REQUEST_URI} (^(admin|cache|misc|modules|sites|system|openid|themes|node/add))|(/(comment/reply|edit|user|user/(login|password|register))$) [OR]
  RewriteCond %{HTTP_COOKIE} DRUPAL_UID [OR]
  RewriteCond %{REMOTE_ADDR} ^192\.168\.28\.12$ 
  RewriteRule .* - [S=2]

  # NORMAL
  RewriteCond %{DOCUMENT_ROOT}/cache/%{HTTP_HOST}%{REQUEST_URI}_%{QUERY_STRING}\.html -s
  RewriteRule .* cache/%{HTTP_HOST}%{REQUEST_URI}_%{QUERY_STRING}\.html [L,T=text/html]
  RewriteCond %{DOCUMENT_ROOT}/cache/%{HTTP_HOST}%{REQUEST_URI}_%{QUERY_STRING}\.json -s
  RewriteRule .* cache/%{HTTP_HOST}%{REQUEST_URI}_%{QUERY_STRING}\.json [L,T=text/javascript]

  ### BOOST END ###
  # Rewrite URLs of the form 'x' to the form 'index.php?q=x'.
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !=/favicon.ico
  RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

</IfModule>

pretty darn close to out of the box I think...

Well, assuming you meant boost/htaccess/boosted2.txt, the cached HTML is bypassed and dynamically created when I substitute those rules, which means it is pretty difficult to know whether the cached CSS and JS were used ... is there any apache debug setting you are aware of that would allow mod_rewrite to log what it is doing?

No, I'm responsible for my own mess I'm afraid :-) I'll look into it though, perhaps there is some lessons there for me.

Thanks for the rewrite pointer, I'll update when I learn more.
Dave

Humour is just so hard to convey with the written word, I'm quite prepared to believe the error is mine not yours! In fact, I've assumed that all along. BUT "mess" is, I guess, the right word in this somewhat embarrassing case.

I do believe it is solved and turned out to be an errant virtual-host file in /etc/apache/sites-enabled which I removed just prior to putting in the rewritelog directive which promptly showed everything is working perfectly. The reasoning is a guess and the fault definitely mine, but the end result is fact.

I am going to get a lot less alerts now.

Your help has been invaluable! I'll be contributing to the mikeytown2 donation bin! Maybe I should go back and review apache docs, it has clearly been way too long!

I took the blame too quickly, it does turn out to be a Drupal thing....if my MAIN site has gzip-enabled CSS and JS per /admin/settings/performance then the cached css/js file is not delivered (I guess my extra virtualhost didn't have anything to do with it...I forgot I made this change, sigh. Sorry for the confusion, trying to work this in on the weekend.

So, the upshot is that the cached CSS and JS files will not be found if the CSS GZIP or Aggregator JS GZIP are enabled ... it looks like there are two engines trying to zip up the files and the rewrite rules can't really deal with the internal redirect to *.gz.

Makes total sense now that I see it written down like that :-)

i reopened this because I was thinking about it and it should be able to work together .. they way things are now I am losing GZIP on my CSS and JS files for authenticated users.

I can attach rewrite logs if you like.

That is correct (though actually, I've chosen to disable GZIP/JS compression preferring the cache instead)

Those look like the same rules I sent you, just rearranged in a different order, is that what you expected?... perhaps my old eyes are missing some change, but regardless it had no effect .... I'll test some more, perhaps my convoluted symlink nightmare (which frankly isn't as bad as all that) caused something to fail but I think not.

Well, with them in place (copy-and-paste from your note), existing unauthenticated users do not get the cached page if the main site (and only the main site) has GZIP enabled....all of the other sites can have gzip enabled but if/when the main site (from default/settings.php) has gzip enabled then the rewrite fails ... the order of the rules does not make any difference. (or the rewrite difference I missed, obviously).

Well, with them in place (copy-and-paste from your note), existing unauthenticated users do not get the cached page if the main site (and only the main site) has GZIP enabled....all of the other sites can have gzip enabled but if/when the main site (from default/settings.php) has gzip enabled then the rewrite fails ... the order of the rules does not make any difference.

After much trial and error I've found the issue ... I think anyway. Parsing those logs is a bit of a pain, I found it somewhat easier to do more directed trial and error. What I've found is that with GZIP enabled for the site then the rewrite request becomes:

sites/default/files/css/css_0056d2dbf1eb4865b7831f5ff92a8817.css.gz

which does not match what is in the boost GZIP cache of:
cache/gz/www.livelink-experts.com/sites/default/files/css/css_0056d2dbf1eb4865b7831f5ff92a8817.css_.css.gz

So, when drupal clears the css cache, the gzipped version goes away and the cached version isn't found. Now that I understand what is going on I can work on rewrite rules that will do what I need; and failing that I have a much better workaround in mind :-)

Hi Mike
Just wanted to give you an update because you seem to take such an interest in support ... I had very little time yesterday to play around, I did quickly try both the patch and the new rules, neither made a difference alone or together -- I still seem unable to serve the _css.gz files. I've no time for the next couple days to look at this, but I will get back to it because I definitely want GZIP to work for both CSS and JS ... those files are huge on my site!

Hi Mike
I finally found the time to do some more digging and you have good news coming I think. What I have verified is that the mod_rewrite is working, boost is working, in fact all is working EXCEPT that my primary server provides the gzipped CSS file to the client in a way that the client doesn't understand it to be gzipped.

Now, the rule in the .htaccess, as you know, is supposed to match %{HTTP:Accept-encoding} !gzip which means that the client accepts gzip encoding, it then looks for a matching file and serves it up... all verified via mod_rewrite logging and a 200 status in access.log but the following URL doesn't work http://www.kinchlea.net/sites/default/files/css/css_75af379721f9360ff83d... ... I am presented with the Drupal "page not found" but using a different server using the same code (NFS mounted) I get the expected CSS content.

The only significant difference I can think of between the two servers is one is the 64-bit version of Ubuntu 9.04 while the other is the 32-bit version... both the same versions (but different compiles) of apache, both same modules installed, same config files. Clearly there is some difference but I'll be damned if I know what it is. I think I may just have to rebuild this server and see if that clears my problems (but then of course I'll never know what the problem was).

In any case, as I said, it can't be boost nor does it appear to be boost's .htaccess rules, so this is now closed.

Well, in the end I'm not sure whether to be proud and happy or embarrassed and happy, but I'm happy just the same!

The problem was the .htaccess created by css gzip in ./sites/default/files/[css|js], when it exists it produces seemingly unpredictable behaviour; most often (I even might go so far as to say always but I thought I had seen different behaviour earlier... I've tried so many incarnations that I am probably mixing them together) the behaviour is that mod_rewrite stops processing the current URI after processing that file regardless of the outcome...essentially making ./sites/default/files/[css|js] the only available caches for CSS and JS files.

Checking off the appropriate no .htaccess radio button in admin/settings/performance and following the directions by moving the rewrite rules into ./.htaccess and poof all is working perfectly. If I had to guess, which I don't but I like to just the same, I'd say it was the recursive use of <ifmodule mod_rewrite.c> blocks but I didn't even try to determine the full story, just happy enough to finally put an end to this niggling problem once and for all.

The reason it looked like different behaviour for different servers was simply because I managed to confuse myself as to which server I was connecting to, while it was up to date and synced vis-a-vis file system, configuration and DB, it wasn't clearing any caches (or indeed running cron at all) so the gzipped files in sites/default/files/ were there to be found by both apache and me (through find(1)).

ANYWAY, while I know my site is uncommon, I am not convinced that this problem is unique to me ... perhaps I'm the only one who hadn't automatically combined the .htaccess files? Doing so was on my future performance tuning todo list but I hadn't really assumed there was any problem to worry about. Silly rabbit, trix are for kids :-)