Comments getting lost on groups.drupal.org

In my instance I began writing a post around 7pm and didn't hit submit until about 2 hours later. My session with g.d.o was still active but after hitting submit the page went directly back to the comment I was replying to instead of back to the main topic.

By getting lost, do you guys mean you left it open in your browser and when you pressed submit it wasn't there, or do you mean it actually submitted...you saw it and then it disappeared. The first is easily explainable. The second is concerning.

-N

I just had a look in the DB. The comment posted in the screenshot (cid 653888) exists in the database and I can see it on the page. I'm not sure I understand the symptoms. More detail would be great. (there are no errors or deadlocks i can see in the DB log)

Thanks.

-N

@jhodgdon,

So the most I can guess at is that groups.drupal.org runs with a limited memcache instance at the moment while we wait for a bug to be fixed. The upshot of this is it could be that you left an issue open long enough that the form token was expired out of the limited cache. This should generate an error, but I've seen it not. (which would mean the issue wouldn't get posted) I'll double check the config when I get back to the US and I'll leave this issue open. I am going to lower its priority though. If anyone else experiences this, please post as much detail here as you can.

Thanks!

I'm watching this but don't have ideas. If it's just limited to that one thread that could be interesting. The only examples I've heard are from that thread, but that could be coincidence.

drumm updated a ton of modules recently and moved it to the more standard bzr/jenkins/settings file setup, which is likely the root cause.

I looked in watchdog and there's no mentions of comment deletes (except a double post I deleted), so this does truly appear to be caused by code/db/memcache issue and not someone maliciously deleting.

If this does happen again feel free to get in touch with me more directly (720-310-5623) - maybe researching immediately after it happens will spark more clues (in watchdog).

nnewton and I discussed a bit. g.d.o is on an outdated version of memcache and we plan to upgrade later this week - that could potentially cause the comments to have never been posted (6.x-1.5, which is not as bad as the 6.x-1.6 through 6.x-1.8 era memcache). It's also possible that you were both drafting responses while drumm did the updates and you lost your sessions/tokens in that process and the comments were never posted.

I haven't, so far, heard of examples outside of that threat which is pretty weird. One commonality of that thread is that people were likely to take a long time in writing their responses which is likely to lead to form cache entries getting lost (via one of the ways mentioned above or maybe something else).

There's a smaller and harder to understand chance that the comments were posted but then somehow deleted/lost.

I can confirm I wasn't working on the site at the same time. I don't have the IRC logs or exact memory, I think jhodgdon and I were chatting about API issues I was reviewing, and I think I was doing something else unrelated too. I did have some good coffee.

I did do upgrades to token and messaging modules later in the day, hoping to solve #1379362: Emails coming from Anonymous.

In my case I didn't receive an error (dsm) or anything I could reproduce. My guess is that the form token expired due to the length of time between when I started the comment and when I actually submitted it. However, I didn't capture enough data to tell for certain.

I took a look at the watchdog logs, but they only go back to December 24 at this time. What's weird is there are 20,012 rows in the table, and http://groups.drupal.org/admin/settings/logging/dblog says we should have 100k.

The latter is due to mysql aut_increment settings and a core bug which probably exists somewhere.

I increased the number we'll log by a 0 so we should get 200,000 now which seems decent. I asked nnewton about syslog and while it's on the roadmap it's currently postponed.

thanks @jhodgdon - mailed him.

Increments of 5. I noticed when rearranging g.d.o/drupal-association that nodes increment by 5 instead of 1. I guess this is what Gerhard is referring to.

@jredding correct. I changed the number of entries to keep in the log from 100,000 to 1,000,000 (i.e. by 0) so I'm hopeful it will add a tens place to the number of entries that are left.

The symptoms from the most recent comment getting lost are helpful:

Logged in, I replied to the thread. I did not get an error message. I may have selected 'save' before 'preview' - not sure if preview is required but did not get a message saying so. When the page reloaded the subject and content fields were empty. I went back to the thread and the reply was not there.

Sounds like memcache form bin getting prematurely emptied. We'll upgrade memcache when nnewton gets out of london stupor and that should either solve it OR we'll remove that cache from memcache.

Fyi - I think I had the same thing described in comment 2 happen earlier today on http://groups.drupal.org/node/197558#comment-658303 - i probably had the tab open for an hour, and when I submitted the comment, it went back to the original comment and the reply I was adding disappeared.

Really sounds as if the forms cache would get purged inbetween. the expulsion rate of the memcached bin should be checked.

the expulsion rate of the memcached bin should be checked.

The forms cache is no longer in memcache as of last week.

This is in settings.local.php:

$conf['memcache_bins'] = array(
  'cache' => 'default',
  'cache_form' => 'database',
);

It seems that either it's getting worse or people are reporting it more :(

Maybe we should just disable memcache altogether?

We're on pretty recent versions of code that should make a difference (e.g. same pressflow as d.o, same memcache as d.o).

@greggles -- has views been updated? We cannot send the full cache back to the DB until the views caching "leak" is fixed in g.d.o. It takes up too much space on our database servers. If that has been updated, I'm fine with putting you back on the DB cache to test if that is related.

-N

Quick update: views had been updated and memcache is no longer being used (afaik). The problem is still coming up though it seems to be less frequent (it was always hard to pin down the frequency so maybe people are just getting used to it).

I think everyone is feeling out of ideas on how to debug, so any new ideas are very welcome. Also increasing the priority and tagging.

I just found http://drupal.org/node/1413566

It's possible that's related (also possible not...). I'm having trouble reproducing that issue, so if others could try it out that would be a great help. You can post dummy content to http://groups.drupal.org/sandbox if you want (though please delete it shortly after posting).

One new module that we could consider (and which would improve user experience in general) http://drupal.org/project/form_save_state

Anyone want to test it out?

A general code review is the first step. Beyond that - I think opening a new tab and logging out/back in and seeing if it saves at least something would be good.

Just almost lost a comment on g.d.o again. When I hit save - saw empty reply form. Hit Back in browser and luckily my comment was still there. When tried to save it again - "Validation error, please try again. If this error persists, please contact the site administrator."

Moving

No reports on this in 2 years. Is it still happening to people?