Rebuilt well over a dozen testbot /tmpfs databases this weekend ... this is not a sustainable process going forward.

We need to i) identify the various scenarios which can result in /tmpfs filling up, ii) determine what exactly is happening behind the scenes, and iii) Code around it so that the bot can detect the issue and fail gracefully before mysql goes away.

Comments

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

Wow, no smoking gun of what patch or whatever might have been doing it?

One thing that's *definitely* happening is that the command you put in to clean up the simpletest tables between tests doesn't always work. That could just be because mysql has died.

We *could* have a root rm -r of that database before beginning a test. Ugly, insecure, but could be done with a setuid.

jthorson’s picture

jthorson commented

Well, there were about 3 patches causing issues ... I know sun's taxonomy-config and crell's drupal-kernel were two of them.

I'd agree with your analysis of the simpletest tables issue ... the cleanup is probably due to mysql going away, after which the whole testbot state is unreliable.

I actually encountered an even more fun one ... twice, the database went away in the middle of a cron run execution. In this case, when you fix the drupaltestbotmysql table and attempt a re-confirmation, the bot refuses to pick up the test. As it turns out, Drupal's cron run sets the cron_semaphore variable in the database when it starts executing, and this never gets reset ... it remains 'TRUE' even after the cleanup, so cron thinks it's already running, and refuses to execute until the variable is manually reset.

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

Just FYI, I doubled memory and tmpfs in the testbots in #1597416: Increase testbot memory

It's not an actual fix, but may affect the behavior for the time being on many patches.

jthorson’s picture

jthorson commented
Status: Active » Fixed

Rendered moot by http://drupal.org/node/1565718, which found the root cause! :)

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

YAY!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.