Hi,

I just released a workflow tarball, but it seems the packing script has choked on it. When I download the tarball (http://ftp.osuosl.org/pub/drupal/files/projects/workflow-4.7.x-1.0.tar.gz) and try to untar it I get:

12:26:50 [mark in tmp]$ tar xfz workflow-4.7.x-1.0.tar.gz 

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error exit delayed from previous errors

Running gzip directly returns a similar error.

12:27:53 [mark in tmp]$ gunzip workflow-4.7.x-1.0.tar.gz 

gunzip: workflow-4.7.x-1.0.tar.gz: not in gzip format

Thanks for your help,
-M

Comments

mfredrickson’s picture

mfredrickson commented
Title: Packing script creates corrupted file » Release link present, file not found
Priority: Critical » Normal

Ok. I'm at least 1/2 an idiot. Curl didn't tell me that I got a 404 on the file. Which I would consider a bug since the release node is published to the public, but the file isn't available.

I'm guessing this is just a timing issue (i see reference that the packaging scripts run 2x/day in some places and every 5 minutes in other places).

Downgrading to see if time heals this wound.

mfredrickson’s picture

mfredrickson commented
Component: Releases » User interface
Priority: Normal » Minor

Ok. The packaging script did eventually run.

I am keeping this bug open and moving it to user interface because I think the information in the drupal_set_messages conflicts. In one case it says packaging scripts run every 5 minutes, in another it says twice per day. Additionally, a link to my release node was present, but the tarball itself was not.

In any case, this is a low priority issue.

dww’s picture

dww
we/he/they
commented

I am keeping this bug open and moving it to user interface because I think the information in the drupal_set_messages conflicts. In one case it says packaging scripts run every 5 minutes, in another it says twice per day.

this is by design. official releases from tags happen every 5 minutes. development snapshots from the end of branches happen every 12 hours. this is documented in many places.

Additionally, a link to my release node was present, but the tarball itself was not.

can you clarify where this link was showing up? where is the release node being linked to or published before the tarball was generated? that shouldn't be happening, unless you're a d.o site admin, and manually published it when you added it (which you shouldn't have done -- the packaging script does that for you once the tarball is ready).

mfredrickson’s picture

mfredrickson commented

Here are some more details:

Project: Workflow
Release: 4.7.x-1.0

Behavior: Immediately after submitting the release node, it was not visible to other users (I checked by logging out and looking at the page). After several minutes the release node did become visible and it was listed on the project page. The listing also included a download link, which pointed to file on ftp.osuosl.org that didn't exist. That file exists now.

I hope this is clearer.

Thanks.

(BTW- I really like the new release system. It took me a little time to wrap my head around it, but it's a godsend in the long run)

dww’s picture

dww
we/he/they
commented
Assigned: Unassigned » dww
Priority: Minor » Normal

ahh, gotcha. this is a new problem, care of http://drupal.org/node/106246

sadly, we don't directly control how often the files are sync'ed between the drupal.org servers and ftp.osuosl.org. i just opened a support request with them to change this. hopefully this will get resolved soon, since there's no good way for the project.module itself to know if the files are on ftp.osuosl.org yet or not.

thanks for the report!
-derek

pwolanin’s picture

pwolanin commented

don't know if this is the same issue but at the time i'm posting, for this project:

http://drupal.org/project/pearwiki_filter

A 5.x dev release is listed in the table, but the timestamp is 0 (i.e. 1969), no release are listed at the "view all releases" page, and no downloadable file is linked.

pwolanin’s picture

pwolanin commented

note, the files were just committed today: http://drupal.org/cvs?commit=49999

dww’s picture

dww
we/he/they
commented
Priority: Normal » Minor
Status: Active » Postponed

this is basically resolved now. ftp.osuosl.org is rsync'ing the tarballs from drupal every 5 minutes now, so the window where this bug is visibile is now quite small.

i'd still like to close it entirely by having our packaging script trigger the rsync before the release node(s) are published, but that's blocked on some server configuration stuff from osuosl. so, i'm downgrading to minor and postponing this, until we hear back from osuosl. in case anyone happens to care, it's [support.osuosl.org #2229] need to modify rsync between drupal2 and ftp.osuosl.org with the osuosl's support tracking system (RT).

cheers,
-derek

dww’s picture

dww
we/he/they
commented
Project: Project » Drupal.org infrastructure
Version: 4.7.x-2.1 »
Component: User interface » Packaging

this no longer has anything to do with the project.module. it's entirely blocked on OSUOSL and d.o infra stuff...

moving to the more appropriate queue in the vain hope others will see it before submitting duplicates.

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented
Priority: Minor » Normal
Status: Postponed » Active

This is not just an issue on some external mirrors -- it is happening on drupal.org. If you create a new release, the new release shows up on your project page on drupal.org as the current release, with a download link, a couple of minutes after you create the new release.

But the delay is not long enough -- the download link itself is not valid until several minutes later -- and if you click on it too soon, you get a 404. Then a few minutes later, the download file is made, and the link is OK.

I have seen this happening several times in the last two weeks. See #599272: Release shows up before download is available, which dww marked as a duplicate of this issue, and #593450: Version 2.2 unavailable, which one of my module users reported against my module, and which I marked fixed because by the time I checked, the link worked.

It would be good to fix this issue.

dww’s picture

dww
we/he/they
commented
Assigned: dww » Unassigned
Status: Active » Postponed

@jhodgdon: It would be good to read this issue before you make (false) assertions about it. We can't reliably fix this issue until the OSUOSL (who hosts drupal.org, in case that's not common knowledge) makes some changes such that we can trigger the rsync that moves the code from where it's created to where it's hosted for download at ftp.drupal.org. No where in this issue are we talking about "some external mirrors". We're talking about drupal.org itself...

jhodgdon’s picture

jhodgdon
she/her
Primary language English
commented

Sorry! I did read the issue, but obviously not carefully enough.... I didn't realize OSUOSL was drupal.org's host (no, it's not common knowledge), adding to my misunderstanding.

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

I have marked #926998: 404 Not Found - for downloading updates - Apache Server at ftp.drupal.org Port 80 as duplicate of this report.

dww’s picture

dww
we/he/they
commented

First of all, re-reading this issue... holy crap do I owe jhodgdon an apology! No idea what stressful badness I was going through when I wrote comment #11 but wow was that uncalled for. :/ Sorry!

Okay, now that that's been said...

We could potentially change the packaging scripts so that they don't publish release nodes at all, and then write some drush script invoked via jenkins that queries for unpublished releases, polls ftp.d.o to see if the files are a 404, and once they're finally a 200, we publish the release (assuming it's not marked as a security update). Seems like a lot of polling and extra churning, but given how often this problem is reported as a new critical bug, etc, maybe it's worth investing the effort and eating the costs of this. It won't be bullet-proof given the ftp.d.o mirrors, but at least it'd be better than what we have now.

Of course, it'd be so much easier if we could just trigger the rsync directly ourselves...

nnewton’s picture

nnewton commented

Hi Derek,

Let me think about this a bit. I don't want us to engineer solutions that we cannot use later on. Right now we are oddly integrated with our mirrors, our DC has direct access and are very open to suggestions. This is not normal and later on in the projects life, I'd not be shocked if we have a larger mirror network and little to no impact on how it runs. (i.e. the sync time maybe longer/different between servers and http-checking each one might become problematic)

-N

dww’s picture

dww
we/he/they
commented

Right, there's no good way to handle this in the general case of totally distributed mirrors, which is what I was hinting at with "It won't be bullet-proof given the ftp.d.o mirrors, but at least it'd be better than what we have now.". But yeah, by all means, give it some thought. There are a lot of things the packaging scripts could be smarter about, especially when it comes to using queues instead of polling (e.g. #381956: Mark project releases as "to be updated" only if pushes happen there). So, we could potentially make use of our queue infrastructure to deal with stuff like this instead of gobs and gobs of more polling.

eliza411’s picture

eliza411 commented
Status: Postponed » Closed (won't fix)

Closing old issues. Please re-open if needed.tag1 github