Testbot broke! What do I do now?

Step 1: Diagnose the problem

In what manner did testbot break?

Testbot says my patch failed

How to tell: Your patch comes back from testbot with an error.
What it means: At some point during the process of applying your patch, installing Drupal from scratch, and running the automated test suite, testbot encountered problems.
What to do: First, read the error message and see if it makes sense in the context of what your patch was trying to do. If so, the problem is with your code, not with testbot. Next, try running SimpleTest module locally and see if you get the same failures. If you do, the problem is with your code, not with testbot. If you do not get the error locally, this may be a know random test. To decide if this is a random failure follow the instructions in 'How to deal with a random test failure'.

If all of those fail to resolve the issue, ping folks in #drupal-contribute with what you're seeing and ask for some advice.

HEAD is failing

How to tell: https://www.drupal.org/node/3060/qa reports a red status
What it means: It could mean that a bad patch got committed to core, or it could mean that there was a temporary glitch with one of the bots
What to do: First, read the error message and see if it correlates to any recent commits at http://cgit.drupalcode.org/drupal/log/. If so, escalate the issue to critical, and leave a comment recommending a rollback of that patch, pointing out the error that testbot is reporting. If not, then you should ping one of the Testbot Admins so they can trigger a branch re-test to eliminate the possibility of a random fail or a bad bot.

The Testbot queue is impossibly backed up

How to tell: Tests are listed as "Queued" or "Running" for many hours.
What it means: It could mean that bots are broken and have been taken out of rotation so there's not enough to handle the load, or it could mean that some bots are "stuck" or [other]
What to do: Ping the Drupal Association infrastructure team to investigate (see below for details).

My patch is not being tested

How to tell: The patch is uploaded to the issue as a file, but there are no drupalci indicators associated with it
What it means: Either the maintainer of the project has not enabled testing for their project, or you need to select the "Needs Review" status on the issue to tell the testbot its ready for testing.
What to do: Contact the project maintainer to enable issue testing, or, set the issue to needs work.

Step 2: Notify the right people

Testbot admins

If you need to trigger a core re-test (e.g. HEAD is failing), go to #drupal-contribute and ping timplunkett, GaborHojtsy, webchick ... [other people with this permission], detailing the problem.

Infrastructure team

If the problem's with testbot infrastructure itself (e.g. the testbot queue is extremely backed up), join the #drupal-infrastructure channel on Drupal Slack and ask if any of the following users are around:Mixologic, drumm, nnewton, or hestenet (in that order), detailing the problem. If no response in ~15 minutes, try e-mail instead.