New project releases not packaging?

No download links are available on this project page.....
Whats the problem?

No download links are available on this project page.....
Project Page URL : http://drupal.org/project/node_revision_delete
Whats the problem?

Seems lots of projects are having this issue. I have a new release of OpenPublish that won't build. Seems the build system is either majorly backed up or crashed.

Edit: Link to my release that isn't built - http://drupal.org/node/1783650

Webform Email Raw, 7.x-1.0-alpha2 has been hung since Sep 13 12:13a ET - no tarballs built, node unpublished:

https://drupal.org/node/1783150

There was a 6.23 to 6.26 upgrade on d.o in the last ~36 hours that may have caused this. The security newsletters were broken for a bit yesterday until drumm found and fixed http://drupalcode.org/project/drupalorg.git/commit/90ecf3ceae3ff9b547e7d...

My guess is that something similar happened with packaging.

Taking this. I think there are two or more problems.

First, one of our packaging scripts, http://localhost:8080/job/package_release_nodes_tag/ hung for the last 13 hours. I kicked it and it is catching up. I expect it to complete in maybe 10 minutes.

That build completed normally, release nodes should be packaged now.

The other issue I noticed is with recommended release calculation, which I'm testing on staging.

I believe this is all fixed up now.

Yes this has been fixed

Same happened again, today.

http://localhost:8080/job/package_release_nodes_tag/223427/console

package_release_nodes_tag hung again today for 18 hours.

This started happening again about 20 hours ago. A remote git clone is hanging.

$ sudo ps -AHf
UID        PID  PPID  C STIME TTY          TIME CMD
bender    9559  9557  0 Nov28 ?        00:00:00               /usr/bin/php /var/www/drupal.org/project/scripts/package-release-nodes.php tag
bender   13488  9559  0 Nov28 ?        00:00:00                 sh -c /usr/bin/php /var/www/drupal.org/tools/drush5/drush.php --no-cache --include=/var/www/drupal.org/tools/drupalorg_drush make --drupal-org=contrib --drupal-org-build-root=/var/tmp/tag.9559/trainingwheels --drupal-org-log-errors-to-file --drupal-org-log-package-metadata=metadata-contrib.json drupal-org.make . 2>&1
bender   13489 13488  0 Nov28 ?        00:00:00                   /usr/bin/php /var/www/drupal.org/tools/drush5/drush.php --no-cache --include=/var/www/drupal.org/tools/drupalorg_drush make --drupal-org=contrib --drupal-org-build-root=/var/tmp/tag.9559/trainingwheels --drupal-org-log-errors-to-file --drupal-org-log-package-metadata=metadata-contrib.json drupal-org.make .
bender   15156 13489  0 Nov28 ?        00:00:00                     sh -c git clone https://github.com/fgnass/spin.js.git /tmp/drush_tmp_1354141294_50b68e6e2c71a/spin 2>&1
bender   15157 15156  0 Nov28 ?        00:00:00                       git clone https://github.com/fgnass/spin.js.git /tmp/drush_tmp_1354141294_50b68e6e2c71a/spin
bender   15158 15157  0 Nov28 ?        00:00:00                         git-remote-https origin https://github.com/fgnass/spin.js.git

$ sudo /usr/sbin/lsof -p 15158
COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
git-remot 15158 bender  cwd    DIR      253,2     4096     362993 /var/tmp/tag.9559/trainingwheels/trainingwheels
git-remot 15158 bender  rtd    DIR      253,0     4096          2 /
git-remot 15158 bender  txt    REG      253,3  2079117    1840491 /usr/local/libexec/git-core/git-remote-https
git-remot 15158 bender  mem    REG      253,0   144776     458996 /lib64/ld-2.5.so
git-remot 15158 bender  mem    REG      253,0  1718232     459216 /lib64/libc-2.5.so
git-remot 15158 bender  mem    REG      253,0    23360     459294 /lib64/libdl-2.5.so
git-remot 15158 bender  mem    REG      253,0   145872     459284 /lib64/libpthread-2.5.so
git-remot 15158 bender  mem    REG      253,0    85544     459225 /lib64/libz.so.1.2.3
git-remot 15158 bender  mem    REG      253,3   204600     460496 /usr/lib64/libidn.so.11.5.19
git-remot 15158 bender  mem    REG      253,0    95464     459380 /lib64/libselinux.so.1
git-remot 15158 bender  mem    REG      253,0   247496     459379 /lib64/libsepol.so.1
git-remot 15158 bender  mem    REG      253,3   252456     459999 /usr/lib64/libcurl.so.3.0.0
git-remot 15158 bender  mem    REG      253,0  1367232     459389 /lib64/libcrypto.so.0.9.8e
git-remot 15158 bender  mem    REG      253,0    92816     459387 /lib64/libresolv-2.5.so
git-remot 15158 bender  mem    REG      253,0     9472     459085 /lib64/libkeyutils-1.2.so
git-remot 15158 bender  mem    REG      253,0    10096     459388 /lib64/libcom_err.so.2.1
git-remot 15158 bender  mem    REG      253,3   613928     460619 /usr/lib64/libkrb5.so.3.3
git-remot 15158 bender  mem    REG      253,3   190976     460665 /usr/lib64/libgssapi_krb5.so.2.2
git-remot 15158 bender  mem    REG      253,3   153720     460609 /usr/lib64/libk5crypto.so.3.1
git-remot 15158 bender  mem    REG      253,3    35984     460608 /usr/lib64/libkrb5support.so.0.1
git-remot 15158 bender  mem    REG      253,0   144344     459303 /lib64/libexpat.so.0.5.0
git-remot 15158 bender  mem    REG      253,0   315080     459390 /lib64/libssl.so.0.9.8e
git-remot 15158 bender  mem    REG      253,0    56752     459238 /lib64/libnss_files-2.5.so
git-remot 15158 bender  mem    REG      253,0    23736     459044 /lib64/libnss_dns-2.5.so
git-remot 15158 bender    0r  FIFO        0,6      0t0 2981500861 pipe
git-remot 15158 bender    1w  FIFO        0,6      0t0 2981500862 pipe
git-remot 15158 bender    2w  FIFO        0,6      0t0 2981500827 pipe
git-remot 15158 bender    3u  sock        0,5      0t0 2981274130 can't identify protocol
git-remot 15158 bender    4u  sock        0,5      0t0 2981274133 can't identify protocol
git-remot 15158 bender    5u  IPv4 2981274137      0t0        TCP util.drupal.bak:53977->www6.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    6u  IPv4 2981274138      0t0        TCP util.drupal.bak:39651->www5.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    7u  IPv4 2981274142      0t0        TCP util.drupal.bak:47330->www7.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    8u  IPv4 2981274392      0t0        TCP util.drupal.bak:52907->www1.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    9u  IPv4 2981274402      0t0        TCP util.drupal.bak:45708->util.drupal.bak:11311 (ESTABLISHED)
git-remot 15158 bender   10u  sock        0,5      0t0 2981274404 can't identify protocol
git-remot 15158 bender   11u  IPv4 2981274414      0t0        TCP util.drupal.bak:41895->www2.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender   12u  IPv4 2981500874      0t0        TCP util.drupal.org:43029->github.com:https (ESTABLISHED)

I filed #1854290: Add timeout to git operations to see about mitigating this in drush make.

From #1854290: Add timeout to git operations

git config has

http.lowSpeedLimit, http.lowSpeedTime: If the HTTP transfer speed is less than http.lowSpeedLimit for longer than http.lowSpeedTime seconds, the transfer is aborted. Can be overridden by the GIT_HTTP_LOW_SPEED_LIMIT and GIT_HTTP_LOW_SPEED_TIME environment variables.

We should set these in git config for bender on util. I'm not sure if environment variables would work or not since there are so many layers of subprocesses.

I'm seeing this trying to package a beta of BlogAPI_New: http://drupal.org/node/1872994 It has now been almost eight nine hours since I originally created the release node.

I reset this again. This also hung on https://github.com/fgnass/spin.js.git. Packaging will take a few minutes to catch up.

@drumm: Thanks!

I guess this was related to the github outages recently https://status.github.com/messages

Hung on spin.js again:

bender   32707 32706  0 10:04 ?        00:00:00                       git clone https://github.com/fgnass/spin.js.git /tmp/drush_tmp_1356516277_50dacbb571a5a/spin
bender   32708 32707  0 10:04 ?        00:00:00                         git-remote-https origin https://github.com/fgnass/spin.js.git
bender     315 32708  0 10:04 ?        00:00:00                           git fetch-pack --stateless-rpc --lock-pack --thin --no-progress https://github.com/fgnass/spin.js.git/  efs/heads/gh-pages
bender     359   315  0 10:04 ?        00:00:00                             git index-pack --stdin --fix-thin --keep=fetch-pack 315 on util.drupal.org --pack_header=2,558

http://localhost:8080/job/api.staging.devdrupal.org%20update%20src/ tries to work around this with

export GIT_HTTP_LOW_SPEED_LIMIT=1
export GIT_HTTP_LOW_SPEED_TIME=60

but it has not been effective. I think we should try setting http.lowSpeedLimit and http.lowSpeedTime in /etc/gitconfig on at least util and stagingwww, in puppet. The actual timeout should be long to allow slow servers to respond, but less than MySQL connection timeouts so that is not lost while waiting for Git.

Stuck on spin.js again. #21 has not been implemented.

Killed another build process. This time it broke because this release https://drupal.org/node/1535950/release tried to download a file that doesn't exist.

Hung on http://git.drupal.org/project/tmgmt.git this time.

Hung on http://git.drupal.org/project/tmgmt.git again.

We haven't tried my idea in #21. I'm not sure where that configuration should live in Puppet.

I have stuck project: http://drupal.org/project/composer_vendor
The 7.x-1.0 release has been created but the package stuff hasn't happend.

killes fixed this just now again.

http://drupal.org/node/1923158 (http://drupal.org/project/complete_profile) is stuck today. :/

re #30 - I think it packaged fine. Just takes a while. The release tag process now takes ~7 minutes and runs every 5 minutes so there's a max of 12 minutes from tagging to creation.

I actually had to stop a process. I've deleted the release that tried to package a non-existing release of something.

@greggles: It was over half an hour when I had last checked and it still hadn't been packaged/published. Maybe I was delirious and it actually wasn't that long. Either way, I gave up and went to bed rather than waiting for it. :)

My new alpha release of http://drupal.org/project/commerce_tnt is not packaging.

http://drupal.org/node/1924256 < this is the alpha release.

Any ideas. Thanks.

Hung on http://git.drupal.org/project/tmgmt.git a couple times this weekend.

I added

[http]
   lowSpeedLimit = 1
   lowSpeedTime = 60

to util's /usr/local/etc/gitconfig, we'll see if it works.

Still hanging on that project.

The issue that is reported in the jenkins output is about patching using

https://drupal.org/files/date_migrate_example_no_version_for_git_deploy-...

and others :(

http://drupal.org/project/cloudwords is stuck. It's been since last night.

http://drupal.org/node/1927686

Stuck on http://git.drupal.org/project/entityreference.git this time. Killing that and it will catch up in the next few minutes.

Stuck on git-remote-http origin http://git.drupal.org/project/entityreference.git again

Seems to hang again, created a -dev for http://drupal.org/project/past yesterday and it's not yet created: http://drupal.org/node/1932976

What's so special about tmgmt and entityreference? (I'm the maintainer of tmgmt, so if we're doing something wrong, I'd like to know :))

Sadly it hung again.

I don't think there's anything special about tmgmt.

guess this is duplicate? http://drupal.org/node/1936938

Any chance this concerns also dev releases? The Title 7.x-1.x dev release is pointing to an outdated tarball. There have been a few commits since then.

Is there a problem with the packaging script today? I created a release last night and it does not seem to have fired yet. http://drupal.org/node/1971334

I may have done something wrong but thought I'd post here too.

It got hung up on grammar parser this time.

Is this still a problem? My release has been hung up for past 24 hours @ http://drupal.org/node/1972380/release

The same thing is happening to me. I even created a new tag to see if it changed anything.
https://drupal.org/node/1079612/release

No tarballs are created at all.

Releases are not being published for http://drupal.org/project/persona

Same for me here with a new dev release http://drupal.org/project/edge_suite

I've restarted the packaging jobs.

In the future, please create one ticket per module or release, this is too difficult to handle. Please always include a link to the release.

Sorry - I did create #1975850: Release node not being published, but I thought it would be better to have a single issue.

Thanks - my releases have appeared!

http://drupal.org/project/openchurch hasn't been working for at least a week now

re #53 - http://drupal.org/node/1524914 and http://drupal.org/node/1334928 point to drush errors that I think you can fix.

@greggles I did fix those, it should have attempted to package again since then.

I think I fixed it. It was a core patch that no longer worked. I wish drush spit out more useful errors but I have this working now, thx!

NVM - Just saw killes post.

this is fixed. Please create new issues for any package not packaging.