Project node release remains unpublished

moving to appropriate queue.

Re: point one about node not showing, it is a known issue. Here is the current workaround: http://drupal.org/node/126554#comment-769513

Re: point two, why can't you use the existing repo for this? If it is just code cleanup, then commit the new code and keep moving. If it is actual CVS structure cleanup then you need to explain what the problem is and how you would like it changed.

@add1sun: I don't think the first issue has anything to do with #126554: cases in edit/releases tab on project nodes: default release broken. The release node itself is unpublished, so that's the reason it's not showing up in the download tables. I could manually publish it, but I decided it was better for dww or someone with more access to chime in here in case this is a bug in the packaging script.

@swentel: As add1sun said, I don't think you need our help for your second issue. However, if you do, please open a separate issue for that, so we don't confuse things in this issue.

I'm experiencing the same problem in trying to create a 1.1 release of Quick Tabs. The release node is here: http://drupal.org/node/236475. I got my colleague Richard, who has admin access, to publish it but there was no download link to the tarball so we just unpublished it again for now.

@nnewton: Is this something you changed with how the packaging script cron jobs are running and the shell configuration on the servers? For example, can drupal-cron still write to the DB? Are these queries going to the non-master DB for some reason, and then getting blasted on DB replication? The packaging script has to update the release nodes once it creates the tarballs to include the link to the tarball, publish the node, update the datestamp and md5 checksum, etc. It appears that there have been some packaging runs that generated tarballs, but nothing was written to the DB (no watchdog entry about it, and no updates to the release nodes).

Quick look at the most recent unpublished release nodes turns up the following:

blockclone 5.x-1.0
booktree 5.x-7.0
quicktabs 5.x-1.1

subscribing

"@nnewton: Is this something you changed with how the packaging script cron jobs are running and the shell configuration on the servers? For example, can drupal-cron still write to the DB? Are these queries going to the non-master DB for some reason, and then getting blasted on DB replication? The packaging script has to update the release nodes once it creates the tarballs to include the link to the tarball, publish the node, update the datestamp and md5 checksum, etc. It appears that there have been some packaging runs that generated tarballs, but nothing was written to the DB (no watchdog entry about it, and no updates to the release nodes)."

I don't think so. I'm still looking into it, but the packaging script uses the drupal.org webroot, which is configured correctly to write to the masterdb. I looked through the binlog on the master and I see updates to the project_release_nodes table, for example today there was an update like this:
#080324 22:40:10 server id 3 end_log_pos 781921114 Query thread_id=121615
4 exec_time=0 error_code=0
SET TIMESTAMP=1206398410/*!*/;
UPDATE project_release_nodes SET file_path = 'files/projects/sf_cache-5.x-1.1.ta
r.gz', file_hash = 'fc651efd1a528411d178936978bace86', file_date = 1206398410 WH
ERE nid = 238296/*!*/;

and later the status of that node is set to 1. I did a check on the slave and its replicating correctly (and that update appears on it).

I looked through the logs for one of the unpackaged nodes (this one: http://drupal.org/node/236475) and see multiple:
DELETE FROM project_release_package_errors WHERE nid = 236475
statements in the log (quite a few actually)...so clearly the packaging script is picking it up and is able to write at least that much to the DB. I'll look looking to make sure something isn't awry, but I doubt its an issue with the DB change.

This is rather odd. These arn't building now because there are newer tarballs already built, but the DB wasn't updated. I tested with quicktabs (moved the tarball out and waited for it to run again) and it correctly built the tarball and updated the node: http://drupal.org/node/236475

I don't really know whats going on here. Is this a continuing problem or are there just these nodes?

Still looking at this...

So take blockclone as an example. The last tarball has the following ctime in seconds since the epoch: 1205965455 (Wed, 19 Mar 2008 22:24:15 GMT). The package release script compares this to the youngest file in the cvs co of the tag. For blockclone, this is 1202246597 (Tue, 05 Feb 2008 21:23:17 GMT). This is causing the youngest check to fail and the release not to be built. I'm thinking these cvs commits failed.

I've unpublished the quicktabs node that I created in my testing of this.

So, at least the quicktabs commit seems to be there, so it may not be that. Also, many of these issues are dated before any of the DB work happened. I got nothing :)

@nnewton: Thanks much for looking into it. Bummer that you "got nothing" after all that detective work. :(

I don't know what to say to the folks experiencing these problems -- the packaging script was written to do the right thing, and seems to have worked flawlessly for over a year. I guess we could just manually fix the few release nodes that seemed to have gotten stuck in this state and hope it was some bizzaro transient failure. But, it's a little unnerving that something which is trying to write to the DB is silently failing... I thought that's one of the things that DBs were supposed to be good for -- keeping track of your attempts to write data. ;)

/me throws up hands and shakes a fist at the gremlins that infest d.o...

I just noticed something. All of these nodes are dated on March 19th between the hours of 1900 UTC and 22-2300 UTC. This is the time period where groups.drupal.org uploaded new code and caused massive load on the webnodes, essentially "DOSing" the infrastructure. This very well could have caused tarballs to be built on this date, without database entries. New release nodes seem to be being built correctly since then. If this could be verified (that release nodes are now building correctly) and the code in the CVS tags verified on these nodes, we could probably just delete these tarballs and get this fixed. Thoughts? (and sorry for spamming this thread)

Sounds utterly reasonable to me. I haven't seen any other (duplicate) reports of release nodes not being published. It does seem to be an isolated, transient failure. If the timing matches, I'm totally happy to chalk this up to more fallout the catastrophic meltdown. I moved the tarballs out of the way and lo, the 5-minute cron job already went off and re-packaged, then published the releases. Let's call this fixed. ;)

Thanks for the investigation, nnewton.