update(_status) results stale for up to 6 hours

This isn't minor -- the module is incorrectly telling me that my installation is insecure!

I think this was a caching bug on drupal.org. I'm no longer able to reproduce this with Drupal 5.3 and Update Status 5.x-2.0.

It can take up to 12 hours for a new release to show up on drupal.org, I believe. 5.3 was released only a few hours ago.

This all goes back to how fancy we are about re-generating the .xml release history files that update_status depends on. Once upon a time, I was going to regenerate them as the packaging script ran, whenever it packaged new releases. That was deemed too complicated, so we just regenerate them all in bulk every 6 hours. However, that makes these situations (especially with security releases) a problem, as update_status is seeing the wrong data for up to 6 hours, and people generate bug reports and support issues every time. :(

So, I think we desperately need to revisit how we re-generate the .xml files, and shrink this window of stale data, at least as relates to SA releases. I see 2 options main options:

A) Provide an admin-only checkbox on release nodes that says "re-generate release history" or something, and have the re-generating script run every 5 minutes, just polling for projects that have release nodes with this bit set. Then, at least for SA-related releases, when we go to publish the release nodes, we can check this box, and at least regenerate the histories for the important projects.

B) Go back to my original design, and have the packaging script keep track of all the projects that it generated any new tarballs for, and then invoke the release-history updater script with that list of projects.

However, there's still a complication with approach (B) as relates to SAs. When originally packaged, any release node that has the "Security release" taxonomy term on it will *not* automatically be published. This gives the security team a chance to verify the tarball, and coordinate its release with the SA going out. It also helps us catch contribs that have "Security releases" that they never told us about and an SA needs to be written, etc. So, that means that at the time of the packaging script invoking project-release-generate-history.php, these release nodes will be unpublished, and that's what's going to show up in the .xml files. Then, some human comes along and eventually publishes the node when we're ready -- we've still got stale info unless we have the manual re-generate override, or we're even more fancy...

C) In addition to (B) described above, in {project_release_nodes} we maintain a "dirty bit", that gets set whenever something about a release node is updated via the Drupal UI. Then, we have a cron job that runs every 5 minutes to query for projects with "dirty release nodes", and regenerate the release history for those and clear the dirty bits. This is sort of a cross between an automated version of (A) and (B). We'd probably still keep something to brute-force regenerate everything every 6 hours (maybe we could up that to 12), just to be safe that nothing slipped through the cracks.

Thoughts? Volunteers?

Thanks,
-Derek

That's an interesting idea, but:

a) That'd involve new versions of update_status for all clients in the wild before the "fix" helps anything, and even then, it's just a band-aid over the problem, not a fix.

b) It doesn't at all solve the more critical problem here: a new security update goes out (a huge part of why update_status exists in the first place) and there's a 6-hour gap before update_status can even start to go into service. :(

So, while it's probably worth looking into your suggestion above as a separate task for the update_status client code, we still need to fix the bug on the server side being discussed here. Furthermore, at this point, I'd rather not tweak the logic for when update_status decides to display an error, since it's somewhat complicated but currently working. ;)

Thanks for the thought...
-Derek

And with the release of 5.4, this needs to be bumped.

I would hope that SAs don't come along that often, so couldn't there just be a 'big red button' that admins would know to press if a SA goes out. And have the packaging script thinger run every 6 hours still.

Hi,

So I recently spent some time looking at this script and the queries it produces. I knocked out the actual write_xml function and ran it against the DB after executing the following on it:

cat project-release-create-history.php | sed 's/\(db_query(\(.*\));\)/ \1 printf(\2); print "\\n";/' > project-release-create-history.php.new

This got me the actual queries generated, which I then took a look at. Not sure what I expected, but they are really excellent queries and its not even worth posting numbers here as pretty much without exception the number of rows looked up per query is under 10 and most of the often repeated ones are well-indexed single row lookups. I am actually quite impressed.

However, (isn't there always one of these) the number of queries is more than mildly concerning. The dumpfile from the script contains on the order of 20 thousand queries. Now I believe they are run in sequence, not parallel, so I am not hugely concerned from a DB perspective, but running that many queries (even very nice queries) is just mildly bothersome. So, I have no technical issue with running this more often...but couldn't we have this either started from the project build script or maybe have a variable in the db set when the build script actually changes something and then have a wrapper function around this script to check that variable?

I see at least once of those options has been brought up before, but I do so love to bring up old issues ;).

How about a hybrid?

We keep a queue table of everything that has been just modified; every 5 minutes cron runs and cleans that queue.

Every 24 hours, we just regenerate everything, to ensure that there is no stale data that got missed somewhere. That allows us to be minorly sloppy with the queue.

But the queue can easily be set up for: adding new releases; editing existing releases; editing the project.

@merlin: Yes, that's exactly what I proposed as option (C) in comment #4 above. ;)

Then there is clearly some agreement at least between the two of us. Slightly different implementation suggested but both are equivalent.

Bump: this is going to be even worse now that update_status 5.x-2.1 and update 6.0-rc3 are out -- see http://drupal.org/node/216212.
I should really move this up near the top (if not at the top) of my d.o plate... We can't have stale XML files anymore, or all hell breaks loose, especially since my proposal for safety-valve code in the client was shot down: http://drupal.org/node/211028.

Bumping with the release of 6.0

Bumping again with 6.2: #244699-1: Update shows Drupal Core 6.1 as "Up to date" (it's not)

well, how about two XML's

update(_status) will first seek in the current releases updates XML, for recent updates, and then looks in the current releases XML for the overall status, ignoring those found earlier.

(it's still a problem...)

Subscribing—above my current pay grade to sort this one out but it does seem like it'd be good to address.

Until this gets fixed, how about upping the frequency of the script? Six hours is a REALLY long time if you just updated one of your modules and then need to push it to a whole bunch of sites (something I do fairly frequently, since I maintain dozens of sites all using all or most of my modules). Luckily in my case, that hasn't been a security issue, but it's definitely a stability issue at times.

I needed this for another Project* site anyway, so I finally got at least part of this working. project-release-create-history.php can now accept a command-line arg to limit itself to a single project. The packaging script now remembers all the projects it touched, and invokes project-release-create-history.php for each one. Doesn't solve the case of manually-published security releases, but it handles everything else. Any objections?

Note: the patch to package-release-nodes.php (184418-19.prn-regenerate-xml.patch) intentionally removes the calls to project_release_check_supported_versions(). I realized that since we're now using node_save(), project_release's own hook_nodeapi() notices that the release is being updated and already calls project_release_check_supported_versions(), so that's just duplicate work now.

I got nnewton to look at this in IRC:

nnewton: dww: k, I think this is fine.
...
nnewton: dww: think we are good, it only regens per project and only those that were built. That seems technically correct
• dww nods
nnewton: I don't think thats going to massively increase load
• dww nods
nnewton: but I'll watch it to make sure
nnewton: let me know when its deployed

So, I committed my patches to HEAD of project/release, and this will go out during tomorrow's d.o downtime.

Now deployed on d.o. Let's call this issues fixed. The remaining edge case for manually published/unpublished releases should be handled over at #548886: Need to regenerate release history XML files when release nodes are manually published/unpublished.

.