Drupal.org
Issues

New project releases not packaging?

Posted by chx on September 13, 2012 at 10:11am
Project:Drupal.org infrastructure
Component:Packaging
Category:bug report
Priority:critical
Assigned:Unassigned
Status:active

Issue Summary

http://drupal.org/node/1783364 ? http://drupal.org/node/1783410 ?

Login or register to post comments

Comments

#1

Posted by kaushalrahuljaiswal on September 13, 2012 at 10:14am

No download links are available on this project page.....
Whats the problem?

#2

Posted by kaushalrahuljaiswal on September 13, 2012 at 10:14am

No download links are available on this project page.....
Project Page URL : http://drupal.org/project/node_revision_delete
Whats the problem?

#3

Posted by jec006 on September 13, 2012 at 2:42pm
Title:node_revision_delete not packaging?» New project releases not packaging?
Priority:normal» major

Seems lots of projects are having this issue. I have a new release of OpenPublish that won't build. Seems the build system is either majorly backed up or crashed.

Edit: Link to my release that isn't built - http://drupal.org/node/1783650

#4

Posted by arpieb on September 13, 2012 at 2:44pm

Webform Email Raw, 7.x-1.0-alpha2 has been hung since Sep 13 12:13a ET - no tarballs built, node unpublished:

https://drupal.org/node/1783150

#5

Posted by greggles on September 13, 2012 at 3:22pm

There was a 6.23 to 6.26 upgrade on d.o in the last ~36 hours that may have caused this. The security newsletters were broken for a bit yesterday until drumm found and fixed http://drupalcode.org/project/drupalorg.git/commit/90ecf3ceae3ff9b547e7d...

My guess is that something similar happened with packaging.

#6

Posted by drumm on September 13, 2012 at 3:56pm
Assigned to:Anonymous» drumm

Taking this. I think there are two or more problems.

First, one of our packaging scripts, http://localhost:8080/job/package_release_nodes_tag/ hung for the last 13 hours. I kicked it and it is catching up. I expect it to complete in maybe 10 minutes.

#7

Posted by drumm on September 13, 2012 at 3:59pm

That build completed normally, release nodes should be packaged now.

The other issue I noticed is with recommended release calculation, which I'm testing on staging.

#8

Posted by drumm on September 13, 2012 at 4:43pm
Status:active» fixed

I believe this is all fixed up now.

#9

Posted by kaushalrahuljaiswal on September 14, 2012 at 8:20am
Status:fixed» closed (fixed)

Yes this has been fixed

#10

Posted by killes@www.drop.org on September 21, 2012 at 2:48pm
Status:closed (fixed)» active

Same happened again, today.

http://localhost:8080/job/package_release_nodes_tag/223427/console

#11

Posted by bdragon on November 15, 2012 at 8:27pm

package_release_nodes_tag hung again today for 18 hours.

#12

Posted by drumm on November 21, 2012 at 12:12am
Assigned to:drumm» Anonymous

#13

Posted by drumm on November 29, 2012 at 7:00pm

This started happening again about 20 hours ago. A remote git clone is hanging.

$ sudo ps -AHf
UID        PID  PPID  C STIME TTY          TIME CMD
bender    9559  9557  0 Nov28 ?        00:00:00               /usr/bin/php /var/www/drupal.org/project/scripts/package-release-nodes.php tag
bender   13488  9559  0 Nov28 ?        00:00:00                 sh -c /usr/bin/php /var/www/drupal.org/tools/drush5/drush.php --no-cache --include=/var/www/drupal.org/tools/drupalorg_drush make --drupal-org=contrib --drupal-org-build-root=/var/tmp/tag.9559/trainingwheels --drupal-org-log-errors-to-file --drupal-org-log-package-metadata=metadata-contrib.json drupal-org.make . 2>&1
bender   13489 13488  0 Nov28 ?        00:00:00                   /usr/bin/php /var/www/drupal.org/tools/drush5/drush.php --no-cache --include=/var/www/drupal.org/tools/drupalorg_drush make --drupal-org=contrib --drupal-org-build-root=/var/tmp/tag.9559/trainingwheels --drupal-org-log-errors-to-file --drupal-org-log-package-metadata=metadata-contrib.json drupal-org.make .
bender   15156 13489  0 Nov28 ?        00:00:00                     sh -c git clone https://github.com/fgnass/spin.js.git /tmp/drush_tmp_1354141294_50b68e6e2c71a/spin 2>&1
bender   15157 15156  0 Nov28 ?        00:00:00                       git clone https://github.com/fgnass/spin.js.git /tmp/drush_tmp_1354141294_50b68e6e2c71a/spin
bender   15158 15157  0 Nov28 ?        00:00:00                         git-remote-https origin https://github.com/fgnass/spin.js.git

$ sudo /usr/sbin/lsof -p 15158
COMMAND     PID   USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
git-remot 15158 bender  cwd    DIR      253,2     4096     362993 /var/tmp/tag.9559/trainingwheels/trainingwheels
git-remot 15158 bender  rtd    DIR      253,0     4096          2 /
git-remot 15158 bender  txt    REG      253,3  2079117    1840491 /usr/local/libexec/git-core/git-remote-https
git-remot 15158 bender  mem    REG      253,0   144776     458996 /lib64/ld-2.5.so
git-remot 15158 bender  mem    REG      253,0  1718232     459216 /lib64/libc-2.5.so
git-remot 15158 bender  mem    REG      253,0    23360     459294 /lib64/libdl-2.5.so
git-remot 15158 bender  mem    REG      253,0   145872     459284 /lib64/libpthread-2.5.so
git-remot 15158 bender  mem    REG      253,0    85544     459225 /lib64/libz.so.1.2.3
git-remot 15158 bender  mem    REG      253,3   204600     460496 /usr/lib64/libidn.so.11.5.19
git-remot 15158 bender  mem    REG      253,0    95464     459380 /lib64/libselinux.so.1
git-remot 15158 bender  mem    REG      253,0   247496     459379 /lib64/libsepol.so.1
git-remot 15158 bender  mem    REG      253,3   252456     459999 /usr/lib64/libcurl.so.3.0.0
git-remot 15158 bender  mem    REG      253,0  1367232     459389 /lib64/libcrypto.so.0.9.8e
git-remot 15158 bender  mem    REG      253,0    92816     459387 /lib64/libresolv-2.5.so
git-remot 15158 bender  mem    REG      253,0     9472     459085 /lib64/libkeyutils-1.2.so
git-remot 15158 bender  mem    REG      253,0    10096     459388 /lib64/libcom_err.so.2.1
git-remot 15158 bender  mem    REG      253,3   613928     460619 /usr/lib64/libkrb5.so.3.3
git-remot 15158 bender  mem    REG      253,3   190976     460665 /usr/lib64/libgssapi_krb5.so.2.2
git-remot 15158 bender  mem    REG      253,3   153720     460609 /usr/lib64/libk5crypto.so.3.1
git-remot 15158 bender  mem    REG      253,3    35984     460608 /usr/lib64/libkrb5support.so.0.1
git-remot 15158 bender  mem    REG      253,0   144344     459303 /lib64/libexpat.so.0.5.0
git-remot 15158 bender  mem    REG      253,0   315080     459390 /lib64/libssl.so.0.9.8e
git-remot 15158 bender  mem    REG      253,0    56752     459238 /lib64/libnss_files-2.5.so
git-remot 15158 bender  mem    REG      253,0    23736     459044 /lib64/libnss_dns-2.5.so
git-remot 15158 bender    0r  FIFO        0,6      0t0 2981500861 pipe
git-remot 15158 bender    1w  FIFO        0,6      0t0 2981500862 pipe
git-remot 15158 bender    2w  FIFO        0,6      0t0 2981500827 pipe
git-remot 15158 bender    3u  sock        0,5      0t0 2981274130 can't identify protocol
git-remot 15158 bender    4u  sock        0,5      0t0 2981274133 can't identify protocol
git-remot 15158 bender    5u  IPv4 2981274137      0t0        TCP util.drupal.bak:53977->www6.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    6u  IPv4 2981274138      0t0        TCP util.drupal.bak:39651->www5.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    7u  IPv4 2981274142      0t0        TCP util.drupal.bak:47330->www7.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    8u  IPv4 2981274392      0t0        TCP util.drupal.bak:52907->www1.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender    9u  IPv4 2981274402      0t0        TCP util.drupal.bak:45708->util.drupal.bak:11311 (ESTABLISHED)
git-remot 15158 bender   10u  sock        0,5      0t0 2981274404 can't identify protocol
git-remot 15158 bender   11u  IPv4 2981274414      0t0        TCP util.drupal.bak:41895->www2.drupal.bak:11211 (ESTABLISHED)
git-remot 15158 bender   12u  IPv4 2981500874      0t0        TCP util.drupal.org:43029->github.com:https (ESTABLISHED)

#14

Posted by drumm on November 29, 2012 at 7:32pm

I filed #1854290: Add timeout to git operations to see about mitigating this in drush make.

#15

Posted by drumm on November 29, 2012 at 10:18pm

From #1854290: Add timeout to git operations

git config has

http.lowSpeedLimit, http.lowSpeedTime: If the HTTP transfer speed is less than http.lowSpeedLimit for longer than http.lowSpeedTime seconds, the transfer is aborted. Can be overridden by the GIT_HTTP_LOW_SPEED_LIMIT and GIT_HTTP_LOW_SPEED_TIME environment variables.

We should set these in git config for bender on util. I'm not sure if environment variables would work or not since there are so many layers of subprocesses.

#16

Posted by DamienMcKenna on December 23, 2012 at 10:57pm

I'm seeing this trying to package a beta of BlogAPI_New: http://drupal.org/node/1872994 It has now been almost eight nine hours since I originally created the release node.

#17

Posted by drumm on December 24, 2012 at 12:28am

I reset this again. This also hung on https://github.com/fgnass/spin.js.git. Packaging will take a few minutes to catch up.

#18

Posted by DamienMcKenna on December 24, 2012 at 1:03am

@drumm: Thanks!

#19

Posted by greggles on December 24, 2012 at 2:16am

I guess this was related to the github outages recently https://status.github.com/messages

#20

Posted by drumm on December 26, 2012 at 6:52pm

Hung on spin.js again:

bender   32707 32706  0 10:04 ?        00:00:00                       git clone https://github.com/fgnass/spin.js.git /tmp/drush_tmp_1356516277_50dacbb571a5a/spin
bender   32708 32707  0 10:04 ?        00:00:00                         git-remote-https origin https://github.com/fgnass/spin.js.git
bender     315 32708  0 10:04 ?        00:00:00                           git fetch-pack --stateless-rpc --lock-pack --thin --no-progress https://github.com/fgnass/spin.js.git/  efs/heads/gh-pages
bender     359   315  0 10:04 ?        00:00:00                             git index-pack --stdin --fix-thin --keep=fetch-pack 315 on util.drupal.org --pack_header=2,558

#21

Posted by drumm on December 26, 2012 at 7:18pm

http://localhost:8080/job/api.staging.devdrupal.org%20update%20src/ tries to work around this with

export GIT_HTTP_LOW_SPEED_LIMIT=1
export GIT_HTTP_LOW_SPEED_TIME=60

but it has not been effective. I think we should try setting http.lowSpeedLimit and http.lowSpeedTime in /etc/gitconfig on at least util and stagingwww, in puppet. The actual timeout should be long to allow slow servers to respond, but less than MySQL connection timeouts so that is not lost while waiting for Git.

#22

Posted by drumm on January 15, 2013 at 8:27pm

Stuck on spin.js again. #21 has not been implemented.

#23

Posted by killes@www.drop.org on February 4, 2013 at 12:10am

Killed another build process. This time it broke because this release https://drupal.org/node/1535950/release tried to download a file that doesn't exist.

#24

Posted by drumm on February 10, 2013 at 6:54am

Hung on http://git.drupal.org/project/tmgmt.git this time.

#25

Posted by drumm on February 12, 2013 at 6:41pm

Hung on http://git.drupal.org/project/tmgmt.git again.

#26

Posted by chx on February 12, 2013 at 7:22pm

Should we add a watchdog script?

#27

Posted by drumm on February 12, 2013 at 7:29pm

We haven't tried my idea in #21. I'm not sure where that configuration should live in Puppet.

#28

Posted by _wdm_ on February 20, 2013 at 4:31am

I have stuck project: http://drupal.org/project/composer_vendor
The 7.x-1.0 release has been created but the package stuff hasn't happend.

#29

Posted by greggles on February 20, 2013 at 3:49pm

killes fixed this just now again.

#30

Posted by Dave Reid on February 21, 2013 at 8:12am

http://drupal.org/node/1923158 (http://drupal.org/project/complete_profile) is stuck today. :/

#31

Posted by greggles on February 21, 2013 at 3:13pm

re #30 - I think it packaged fine. Just takes a while. The release tag process now takes ~7 minutes and runs every 5 minutes so there's a max of 12 minutes from tagging to creation.

#32

Posted by killes@www.drop.org on February 21, 2013 at 3:17pm

I actually had to stop a process. I've deleted the release that tried to package a non-existing release of something.

#33

Posted by Dave Reid on February 21, 2013 at 3:30pm

@greggles: It was over half an hour when I had last checked and it still hadn't been packaged/published. Maybe I was delirious and it actually wasn't that long. Either way, I gave up and went to bed rather than waiting for it. :)

#34

Posted by jbloomfield on February 22, 2013 at 2:03pm

My new alpha release of http://drupal.org/project/commerce_tnt is not packaging.

http://drupal.org/node/1924256 < this is the alpha release.

Any ideas. Thanks.

#35

Posted by drumm on February 24, 2013 at 7:44pm

Hung on http://git.drupal.org/project/tmgmt.git a couple times this weekend.

I added

[http]
   lowSpeedLimit = 1
   lowSpeedTime = 60

to util's /usr/local/etc/gitconfig, we'll see if it works.

#36

Posted by killes@www.drop.org on February 25, 2013 at 8:18am

Still hanging on that project.

The issue that is reported in the jenkins output is about patching using

https://drupal.org/files/date_migrate_example_no_version_for_git_deploy-...

and others :(

#37

Posted by twistor on February 26, 2013 at 4:15pm

http://drupal.org/project/cloudwords is stuck. It's been since last night.

http://drupal.org/node/1927686

#38

Posted by drumm on February 26, 2013 at 5:10pm

Stuck on http://git.drupal.org/project/entityreference.git this time. Killing that and it will catch up in the next few minutes.

#39

Posted by drumm on February 28, 2013 at 6:17pm

Stuck on git-remote-http origin http://git.drupal.org/project/entityreference.git again

#40

Posted by Berdir on March 5, 2013 at 10:49am

Seems to hang again, created a -dev for http://drupal.org/project/past yesterday and it's not yet created: http://drupal.org/node/1932976

What's so special about tmgmt and entityreference? (I'm the maintainer of tmgmt, so if we're doing something wrong, I'd like to know :))

#41

Posted by killes@www.drop.org on March 5, 2013 at 11:39am

Sadly it hung again.

I don't think there's anything special about tmgmt.

#42

Posted by pwolanin on March 8, 2013 at 1:46pm

guess this is duplicate? http://drupal.org/node/1936938

#43

Posted by plach on March 19, 2013 at 4:16pm

Any chance this concerns also dev releases? The Title 7.x-1.x dev release is pointing to an outdated tarball. There have been a few commits since then.

#44

Posted by haydeniv on April 16, 2013 at 4:51pm

Is there a problem with the packaging script today? I created a release last night and it does not seem to have fired yet. http://drupal.org/node/1971334

I may have done something wrong but thought I'd post here too.

#45

Posted by drumm on April 16, 2013 at 6:09pm

It got hung up on grammar parser this time.

#46

Posted by sylus on April 18, 2013 at 3:05pm

Is this still a problem? My release has been hung up for past 24 hours @ http://drupal.org/node/1972380/release

#47

Posted by perusio on April 21, 2013 at 1:08pm

The same thing is happening to me. I even created a new tag to see if it changed anything.
https://drupal.org/node/1079612/release

No tarballs are created at all.

#48

Posted by jbrown on April 22, 2013 at 11:17am
Priority:major» critical

Releases are not being published for http://drupal.org/project/persona

#49

Posted by ti2m on April 22, 2013 at 11:25am

Same for me here with a new dev release http://drupal.org/project/edge_suite

#50

Posted by killes@www.drop.org on April 22, 2013 at 11:40am

I've restarted the packaging jobs.

In the future, please create one ticket per module or release, this is too difficult to handle. Please always include a link to the release.

#51

Posted by jbrown on April 22, 2013 at 11:43am

Sorry - I did create #1975850: Release node not being published, but I thought it would be better to have a single issue.

#52

Posted by jbrown on April 22, 2013 at 11:47am

Thanks - my releases have appeared!

#53

Posted by drupalninja99 on May 10, 2013 at 12:26pm

http://drupal.org/project/openchurch hasn't been working for at least a week now

#54

Posted by greggles on May 10, 2013 at 12:58pm

re #53 - http://drupal.org/node/1524914 and http://drupal.org/node/1334928 point to drush errors that I think you can fix.

#55

Posted by drupalninja99 on May 10, 2013 at 1:17pm

@greggles I did fix those, it should have attempted to package again since then.

#56

Posted by drupalninja99 on May 12, 2013 at 11:15pm

I think I fixed it. It was a core patch that no longer worked. I wish drush spit out more useful errors but I have this working now, thx!

nobody click here