Attachments failing when users preview or use "attach" button

This is becoming very annoying - can take up to 5 attempts to attach a patch, some people are having to resort to uploading patches on their own sites and linking to them.

This could be a bug in the project issue module, but I don't think I've ever had this problem on a test site, so it might have something to do with the d.o infrastructure itself.

This is going to be hard to fix unless we have a way to reproduce it on a test site. Some questions that might help us figure out what's going on:
1. Does this happen only on comments or on original issues as well? (This helps to determine if it's a comment_upload bug or not).
2. Do attachments get lost if you click the "Attach" button after selecting the file? Do they get lost if you don't hit the "Attach" button and just submit the issue/comment? Or maybe in both cases.

From the few times I've seen this I think it happens on both comments and original issue nodes, and I think I've seen it happen in cases where I have and have not clicked the "Attach" button first. But I'm not so sure about this second point.

1. I've only had this on comments - although I can't remember the last time I opened an issue with an attachment directly ;)
2. It happens when I hit the attach button and can see the attachment listed, I started double-checking this when I first noticed it.

That's what I was afraid of.

Have either of you ever had this happen when attaching a file to a node of another type on d.o? The project_issue module just uses the core upload module, so there's a possibility the problem is in core and not in project_issue.

Doesn't project_issue also use comment_upload? If the problem occurs in both modules that would seem to point rather to an infrastructure issue.

Since the form itself works and files appear in preview, I also think it points to an infrastructure issue. I've not tried other content types no.

The only other site I've ever seen these problems on was security.d.o, further pointing to d.o infrastructure badness, not a code bug... So, I think "File server" is a more appropriate component than "drupal.org module" (at least until we have more info about what's actually going wrong).

well... http://nerdpalace.net/patches/advf_preprocessor.patch

Please ignore me, I can't seem to handle tabbed browsing today.

Is it possible that the problem is related to this?: [infrastructure] Networking Issues: http://lists.drupal.org/private/infrastructure/2008-June/011678.html

For those without the benefit of access to that private mailing list archive, here's what oadaeh is talking about:

From: Narayan Newton
Date: June 10, 2008 1:01:26 PM PDT
To: Drupal.org Infrastructure Maintainers
Subject: [infrastructure] Networking Issues

Hi All,

As has been reported, we have been having intermittent issues with the
connection between the webnodes and the database backend, as well as
the NFS share between the webnodes. Looking into this today, we have
discovered TCP stream errors and possible dropped segments between the
nodes. We are contacting our networking staff and the infrastructure
lead for the OSL is working on the issue.

My apologies for the inconvenience.

--
Narayan Newton
Database Administrator
OSU Open Source Lab
GA Member Drupal Association

Yes, that seems entirely plausible as a possible cause for these sorts of errors. If the webnodes are having trouble talking to NFS, that would explain why the webnodes think the file is uploaded, but then the attachment never actually exists when you submit your issue or comment...

I have had (more than) one patch disappear that I know had been attached http://drupal.org/node/134949#comment-870466. Also I have seen one disappear, and then reappear after 3 refreshes of the page (not re-submissions) ie. about 25-30s after submission. D.o seemed to be generally under stress at the time, so I wouldn’t be surprised if this turns out to be an infrastructure issue.

The NFS/Networking issues could be causing this. We are annoying the network people here to find out what is going on.

-N

Thanks, nnewton.

Side note: I've gotten in the habit of *not* pressing the "attach" button on d.o, and so far, every one of my comment attachments has gone through. Every time I hit "attach" and let the JS do its magic, the attachment didn't work. The sample size is still pretty small (less than 5 of each). Not sure if that's a useful datapoint, but I thought I'd share in case it's useful for debugging.

Cheers,
-Derek

That is interesting....doesn't sound like the issue I was thinking of. The problem I'm working on fixing appears to be at the switch level at this point.

-N

@nnewton: Could be just random results due to the small sample size. NancyDru reports in #4 that she's seen the bug both with and without pressing attach... I'm still hoping this is an NFS/networking problem.

Each time I post a patch, I just browse to it, don't hit attach, and click 'Post comment' without previewing first. I've never had a patch disappear on me, except for one time when I did a preview first.

As a result, this smells more like a Project bug to me.

@webchick: except...
a) I use project* on a lot of sites, and I've never had this trouble except on the d.o infrastructure.
b) None of the project* developers can reproduce this on local test sites.
c) If it's a bug, it'd be in comment_upload and/or core's upload.module, not project_issue. ;)

As an update, the NFS master has been moved to a different switch card. Please let me know of this happens again.

It just happened to me again. First try failed, and then 2nd try worked. FYI, it was a comment, and I hit 'attach' button both times.

The problem persists. Tested more then once yesterday and today.

Had a couple of uploads go missing yesterday as well.

I don’t know if this is related: http://drupal.org/node/271215

i can confirm that Webchick's #19 is a bulletproof method of attaching patch files to d.o issues. I came to this same conclusion independently, and just wanted to reinforce that this is the most reliable method.

Is there a reason why d.o infrastructure would prefer Webchick’s method #19™ or do you think it is project*/comment_upload related?

I note that both Senpai and I are site maintainers, and NancyDru is not. Could this be permissions-related?

Could this be permissions-related?

I would guess that if it were permissions related, then only site maintainers would be allowed to upload, instead the problem is intermittent. Just a thought.

Another correlation does not imply causation idea: I've noticed other issues raised concerning throttle.module, is the upload.module being switched off on throttle?

I would guess that if it were permissions related, then only site maintainers would be allowed to upload, instead the problem is intermittent. Just a thought.

It's intermittent, but only for certain people. As mentioned, I have yet to run into this bug doing #19 and I upload probably 50 patches a day (ok, not really, but quite a few! :)).

Another correlation does not imply causation idea: I've noticed other issues raised concerning throttle.module, is the upload.module being switched off on throttle?

Throttle module is disabled on d.o. So nope, that's not it either. :(

Same here - doing #19, I've not lost an attachment yet. I'm losing loads if I hit the upload button first though. I'm also a site maintainer.

I've had this issue also, but I thought it was just a permissions problem. I always clicked the Attach button. Won't do that in the future.

Un-assigning this from me. If a specific method always works, this likely isn't NFS...

Just happened to me - I used the attach button.
Following up with the normal submit worked.
A sample size of one so far.

Wanted to add another data point to this. This just happened to me but on a *handbook page*, not in the issue queue. I was trying to attach an image to add to the page. Using the attach button had it disappear while just skipping that and going straight to submit got me set right. Luckily I only had one image for the page. If I'd had multiple I suppose I'd have to edit > browse > submit for each rather than uploading them all at the same time.

Oooo! Interesting data point!

* add1sun is a site maintainer as well.
* uploads on handbook pages have nothing to do with project* module
* This points to a larger issue with either upload.module OR the filesystem.

Here's another potentially interesting data point.

Yesterday I created a new issue (#273523: E_NOTICE in theme_fieldset()). I attempted to attach 2 patches by doing this:

1. Clicked browse to select file A.
2. Clicked the Attach button.
3. Clicked browse to select file B.
4. Clicked the submit button to submit the issue.

After submission, file A was attached but file B was not. I'm not sure if this qualifies as a counterexample to the idea that not clicking the attach button will always result in a successful attachment, because I didn't click the attach button after selecting the *second* file and that's the one that failed to be attached.

In any case I think we can probably rule out that comment_upload is at fault here, since it seems that several people are having problems with attaching files to original issue nodes.

comment_upload uses private functions from upload.module for the heavy lifting, specifically:

http://api.drupal.org/api/function/_upload_prepare/5

Does all kinds of $_SESSION fun, so I reckon the problem is in there.

This was reworked a fair bit for D6 (i.e. these functions don't even exist any more).

Does the loadbancers use sticky sessions? If not, this could really be source of this issue...

Changing the title to be more descriptive. This is an upload module problem and not an "issue" nor comment_upload problem. And my permissions are enough to make me dangerous...it's not a permission issue.

I just created a book page as a test of this. I used 4 dummy text files. I attached the first 3 by using the "browse and click attach button" method. Then I hit preview. Then I browsed to the 4th file without using the attach button and hit preview. Then I hit submit. All 4 text files have disappeared.

I then edited the node and attached file 1 via the attach button method, file 2 via a preview, and file 3 via browsing to the file and hitting submit. Those worked perfectly.

I then deleted the first book page and created a new one following the same steps. This worked perfectly again.

The problem is intermittent, but it doesn't take too long of testing to find it.

Moshe has reminded us that we should have a tmp directory inside files/ so all webnodes can access it. It used to be set up like this, but somehow the setting got lost, probably during the recent problem.

Marking fixed, reopen if the problem reoccurs.

Sound like a missing "requirement" check of a the module that uses the tmp directory!?

I believe paranoia requirements are usually only checked at admin pages - when you visit the admin/filesystem or site status page.
The rest of the time modules are justified in assuming that the system is all as it was last time they looked. Folders going missing mid-flow is an unpredictable and not-to-be expected circumstance. More to the point, running that check every time on the off-chance is too expensive. IMO.

If #43 was indeed the problem, then just opening up the admin panel would have highlighted it and pointed to the easy fix.
Me, I would have suspected session trouble with the load balancers, but I don't actually know enough about the internals to make anything but wild guesses.
#43 says closed, with a plausable fix, so fingers crossed.

Yep, fingers crossed.

@dman: If you have *real* loadbalancing (not sticky sessions) *every* request is load balanced across your server farm and could be distributed to a different webserver machine. This depends on loadbalancer configuration settings :-). In this way you loose control where you are and you cannot access the local server memory between requests. It's possible that a $_SESSION['foo'] is set on server A and $_SESSION['bar'] on server B. So you cannot rely on local webserver memory variables... except you site uses a single central storage in the database that keep both client variables together over request.

I've had 2 patches disappear in exactly this fashion this week. Would someone mind checking if the temp directory got lost again? Thanks!

Nothing new for ~6 months. Hopefully fixed. I haven't seen it anymore myself.