Squid is causing havoc for update_status.module and update.module

AS part of the testing, I had webernet hit the XML file for update_status manually. ON his first hit the -rc did not appear; after that it did.

I personally was unable to reproduce this in at least 20 hits.

I don't know where in the chain the stale data was coming from, but *something* was feeding him stale data somewhere.

I blame squid. :( This is really a d.o infrastructure issue. Maybe there's some way that the thin wrapper script serving the .xml files can set some special cache lifetime headers so squid doesn't do the wrong thing? I don't know nearly enough about how squid is working to fix this myself, but hopefully by bumping this into the infra queue, some other eyes will see it and someone can suggest an appropriate fix.

The bugs just keep getting reported: http://drupal.org/node/157659

This is critical. How can we get squid to stay the hell out of the way for update.drupal.org, or at least, to ensure that the squid cache is always cleared for these URL every 6 hours when the new release history .xml files are generated?

PLEASE? Anyone who understands the squid setup on this list? I'm dyin' here. ;)

Thanks,
-Derek

Check out http://wiki.squid-cache.org/SquidFaq/InnerWorkings

I mentioned the Cache-Control header to you on IRC, the above page specifies where Squid looks at it. Here's the technical read on it:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9

In short, the Squid server will work properly without any additional hand-holding if Drupal is sending the correct Last-Modified HTTP header, and there is no way that there could be a 6-hour delay. If Squid's refresh algorithm isn't working for you (an example would be a page that changes once a week, but when it changes you need to know in 5 minutes), that's when you need to have your page use HTTP headers to tell proxies how to deal with you page. This isn't just about the reverse proxies I set up either: if someone is behind a regular proxy at work or school that does page caches, which is not uncommon, its important to inform these proxies how long to cache your page for.

For example, if the client requesting the page (part of this module) sends a `max-age' value in a Cache-Control line in the HTTP request, the proxy will respect this value by not serving a page older than this value. Or, the response (also part of the same module, I think) can also send the maximum time it can be cached (also `max-age'), or prevent caching at all with `no-cache' (of course, the former with a small time, say 5 minutes, is much preferred, especially for these auto-update pages which are getting frequent hits from sources checking for updates).

If it's reasonably difficult to implement a patch that does this in the short time frame you need it fixed in, it is possible for me for me to insert arbitrary headers at the web-server level depending on things like the URL path. This method, however, does not scale to more than a few URLs that need this, and it would be best for it to only be a short term solution until the code itself supports properly representing its age to proxies (both forward and reverse).

Hope this helps,

-Eric

So, something like this?

Sorry, I guess I should have been clear about this. These .xml files aren't being served by Drupal at all. There's a trivial php wrapper that's doing it. See http://drupal.org/cvs?commit=71890 for details. The patch in the previous comment is against this wrapper script.

The patch appears right to me.

In IRC, emsearcy said to just commit the Last-Modified part, and forget about Cache-Control for now. See http://drupal.org/cvs?commit=72922. Now installed on updates.d.o. Let's see if that solves it.

Thanks, folks!
-Derek

Ugh. I'll try to track down emsearcy or someone else squid-aware and see what more we can do to make sure we're not serving stale data. :(

In IRC, emsearcy recommended setting the max-age Cache-Control header. Just committed this to HEAD (http://drupal.org/cvs?commit=74063) and installed on d.o. Let's see if that does the trick. Please re-open this at the first sign of trouble and we'll keep digging.

Despite several recent updates listed on Drupal for my site's modules, the Available Updates page (from the Update Status mod) hasn't acknowledged them. I've manually run cron jobs several times, but no joy.

For modules that I've updated anyway and after force-checking my Available Updates page, the new module is designated as "Update available" -- the suggested update is the previous version.

Unfortunately, from the infrastructure standpoint I do not have enough information at the moment. In order to know what the issue is here, I need the data right when the problem is occurring :-(. One thing anyone who sees this can do is post the URL that is being fetched to determine the versions for your particular module. You should be able to put this URL into your web browser and see the stale XML version info.

I have one last idea, which is that we still have stale pages because they were originally entered into the cache with a high age, so the recent patches may not affect these. I'm clearing out the caches in a moment.

Keep me posted.

I had another thought about this. As soon as we commit the patch for http://drupal.org/node/128827, we're going to have to make sure squid and other caches do *NOT* cache these pages *AT ALL*. We need to go all the way to the wrapper for each request, or the logging of stats will be skewed and busted.

Since we're going to need to set "Cache-Control: no-cache, must-revalidate" at that point (hopefully soon), why don't we just do that now? Then clear the squid cache once and for all, and leave squid out of the picture...

Sound like a plan?
-Derek

Just committed http://drupal.org/cvs?commit=74854 to disable all caching, both since this continues to cause trouble for folks, and since we need it as soon as http://drupal.org/node/128827 lands.

Can someone who's squid-aware please purge the entire squid cache of anything from updates.drupal.org?

Thanks!
-Derek

OK emsearcy - you asked for more info about this problem. Below is a copy from my site´s "Update status":

----------------------
Available updates List Settings
Here you can find information on the update status of your installed modules. Note that each module is part of a "project", which may have the same name as the module or may have a different name. Also note that this can only check the status for "official releases", and will not be able to check update status for development snapshots and modules updated directly from CVS.

Last checked: Fri, 2007/07/27 - 13:04 Check manually

Project Status Current version Available version Download latest version
Drupal Up to date 5.2 5.2 (2007-Jul-26) drupal-5.2.tar.gz
Includes: Block, Filter, Node, System, User, Watchdog
Javascript Tools Up to date 5.x-0.6 5.x-0.6 (2007-Mar-19) jstools-5.x-0.6.tar.gz
Includes: Javascript tools
Update status Update available 5.x-1.2 5.x-2.0-rc (2007-Jun-28) update_status-5.x-2.0-rc.tar.gz
Includes: Update status
Views Update available 5.x-1.6-beta4 5.x-1.6 (2007-Jul-15) views-5.x-1.6.tar.gz
Includes: Views
----------------------

Drupal and Javascript Tools are shown in green. Update status and Views are shown in red.

Thanks / Tomas

If I read this right, you're running Update Status 1.2? If so, the caching problem won't affect you as you're using XMLRPC.

I recommend updating to the 2.0-rc. It's proven to be quite stable.

I hadn't realized until chatting with wwwebernet on IRC that the older drupal update_module clients we still want to support are directly pulling the XML files, rather than using the newer PHP wrapper. Therefore, the patch against project-release-serve-history wasn't completely solving the problem. It was actually a pretty trivial fix once the problem was explained as such to me, and adding the following to the vhost config (and clearing the squid cache) has seemed to at least put the right headers on the XML files. Hopefully the problems we're seeing will subside now.

+
+ # Fix problem with update module receiving stale data
+ Header append Cache-Control "max-age=600, must-revalidate"
+

Right, unfortunately, the "5.x-2.0-rc" release was directly fetching the .xml files from http://updates.drupal.org/files/release-history :( That was a stupid move on my part, and is fixed in 5.x-2.0-rc2 and the version in D6 core. See http://drupal.org/node/155281 for more.

Luckily, with 5.x-2.0-rc2 out (and in a few days, 5.x-2.0 itself), we should see a (hopefully) quick drop-off of the old clients once everyone upgrades to the official 5.x-2.0 release.

Thanks for your continued help on this, Eric!