Boost not clearing cached content

Discovered that Drupal wasn't picking up the name of the site so I've set $base_url in settings.php. Crawler now seems to be running, but still not picking up fresh edits.

What I should have said in my initial run down was that our live URL is actually attached to a proxy server that proxies requests through to our Drupal server. We have it set like this so we can map other sites located on the proxy server into the same URL. What I'm wondering is whether because the cache is being created under the Drupal server name (which is different from our live URL) whether this is causing it not to pick up the edits?

I've set the ETAG options, so I'll see whether that improves things.

Thanks again for the response, much appreciated.

Our public URL is of the form

www.sub.domain.com

This takes requests, and proxies them through to our drupal server which lives at

drupal.sub.domain.com

drupal.sub.domain.com is our staging server where all the edits take place.

On our staging site, should I cahnge the directory where the cache gets written to to

cache/normal/www.sub.domain.com

Just some further info on how the sync takes place. We have a cron job that is set to run every hour which utilises the drush command 'rsync' and 'sql-sync'. The cron job then remotely runs the cron job on the live server to kick off the crawler.

I've tried unchecking "Do not store the cache file path in the database" and instead forcing both our staging and live server to use the same directory for the cache. I've also updated the Rewrite rules to reflect this as follow.

  # GZIP - Cached css & js files
  RewriteCond %{HTTP_COOKIE} !(boost-gzip)
  RewriteCond %{HTTP:Accept-encoding} !gzip
  RewriteRule .* - [S=2]
  RewriteCond %{DOCUMENT_ROOT}/cache/perm/sitecache%{REQUEST_URI}_\.css\.gz -s
  RewriteRule .* cache/perm/sitecache%{REQUEST_URI}_\.css\.gz [L,QSA,T=text/css]
  RewriteCond %{DOCUMENT_ROOT}/cache/perm/sitecache%{REQUEST_URI}_\.js\.gz -s
  RewriteRule .* cache/perm/sitecache%{REQUEST_URI}_\.js\.gz [L,QSA,T=text/javascript]

  # NORMAL - Cached css & js files
  RewriteCond %{DOCUMENT_ROOT}/cache/perm/sitecache%{REQUEST_URI}_\.css -s
  RewriteRule .* cache/perm/sitecache%{REQUEST_URI}_\.css [L,QSA,T=text/css]
  RewriteCond %{DOCUMENT_ROOT}/cache/perm/sitecache%{REQUEST_URI}_\.js -s
  RewriteRule .* cache/perm/sitecache%{REQUEST_URI}_\.js [L,QSA,T=text/javascript]

  # Caching for anonymous users
  # Skip boost IF not get request OR uri has wrong dir OR cookie is set OR request came from this server OR https request
  RewriteCond %{REQUEST_METHOD} !^(GET|HEAD)$ [OR]
  RewriteCond %{REQUEST_URI} (^/(admin|cache|misc|modules|sites|system|openid|themes|node/add))|(/(comment/reply|edit|user|user/(login|password|register))$) [OR]
  RewriteCond %{HTTP_COOKIE} DRUPAL_UID [OR]
  RewriteCond %{HTTP:Pragma} no-cache [OR]
  RewriteCond %{HTTP:Cache-Control} no-cache [OR]
  RewriteCond %{HTTPS} on
  RewriteRule .* - [S=3]

  # GZIP
  RewriteCond %{HTTP_COOKIE} !(boost-gzip)
  RewriteCond %{HTTP:Accept-encoding} !gzip
  RewriteRule .* - [S=1]
  RewriteCond %{DOCUMENT_ROOT}/cache/normal/sitecache%{REQUEST_URI}_%{QUERY_STRING}\.html\.gz -s
  RewriteRule .* cache/normal/sitecache%{REQUEST_URI}_%{QUERY_STRING}\.html\.gz [L,T=text/html]

  # NORMAL
  RewriteCond %{DOCUMENT_ROOT}/cache/normal/sitecache%{REQUEST_URI}_%{QUERY_STRING}\.html -s
  RewriteRule .* cache/normal/sitecache%{REQUEST_URI}_%{QUERY_STRING}\.html [L,T=text/html]

Both sites are now creating caches in the correct directory, and Boost is definitely working as I can see the boost tags at the bottom of the source in each page. However it still doesn't seem to be expiring content.

What effect would the above changes have on the crawler? Does it know which files to expire given that Boost is now using the same directory and not creating a new one per domain?

Hitting the site directly or via the proxy doesn't seem to get the page regenerated until a CTRL+F5 is used.

Another pondering...

When I'm doing the sql-sync from the staging to live server, I'm copying all the boost tables. So the tables are being populated with URLs that are pointing to my staging server and not the live server. I have updated the drush aliases to exclude these tables and they are now being populated solely by the live site.

Here's a question. How does Boost know that content has changed? When a node is updated, is there a flag generated anywhere for boost to say it's out of date? Or do we just have to wait for the page to expire before it is crawled?

That would make sense. So because I'm not copying across boost_cache, it's not expiring content when the node is updated. Maybe need to create a small script to do that manually...

Just did a test on our staging server and the "expire" column got set to 434966400 when the node was changed.

Hmmm, so I presume running cron does something to this expire timestamp? After running cron, the node is correctly listed as 0 for the expire column in boost_cache.

I'm seeing content in boost_cache that has a timestamp that is less than the current time, so I'm presuming this should get picked up on the next cron run and refreshed. However this is not happening. The crawler is definitely running as I can see other URLs being populated to boost_crawler.

Am I misunderstanding the logic here?

The logic is, as I understand it...

Crawler is kicked off
Checks to see if the site has been changed
- It does this by going through the different tables and getting back the most recent timestamp
- Once it has retrieved these values it compares it to boost_max_timestamp in the 'variables' table
- If it matches, then the site hasn't been changed and false is returned.
- If it doesn't match, then true is returned.
Expires content by setting 'expire' to 0.