Testbot running awfully slowly, anything that can be done to improve it?

Generally, the queue wait time has been running under one hour for most tests since code freeze, with a significant amount of time still being spent idle. On average, it's been busy, but not overloaded.

That said, what can happen occasionally is that the D8 HEAD test gets marked as broken ... either as a false positive, or because it actually is. This was the case yesterday; where testbot marked it as bad on at least four different occasions. While D8 HEAD is in this state, any new D8 patches that get posted are temporarily postponed ... and once the state clears (generally ~ 1 hour later, after the D8 test is run again), they are all dumped into the queue at once. Where the D8 test fails multiple times in a row, this can result in a fairly large backlog of tests building up ... causing the log jam that we've seen a few times in the last month.

There were two other instances of what I'd consider 'abnormal' load recently ... in one case, a large number of core RTBC patches which were sitting in the queue for a week were re-queued in order to re-test them before a large commit streak ... and in the other, a number of year-old 'Needs Review' patches were re-queued to see whether they still applied. Both of these situations caused a significant traffic jam as well.

The ability to prioritize contrib versus core is an issue that's long been on my radar; but we had more or less feature-frozen PIFT/PIFR to focus on the development of the next iteration of testing infrastructure and in anticipation of the Drupal.org D7 upgrade. Since this obviously has not occured as quickly as we had hoped, I've gone back and have been committing new features to testbot once again ... and would gladly help support anyone who wanted to dive into the codebase and try to implement any new features!

That said, and based on my own experience, navigating the code goes much easier when you've got a guide who's familiar with the environment ... so if this is something you'd like to pick up, let me know and I can provide some info to point you in the right direction.

Yep ... there has been some trouble with the jenkins server which runs the 'queue-ing' jobs ... it's been getting backed up, and the drupal.org <-> qa.d.o communications is one of the items which becomes affected as a result.

And to address part of the original post, the plan for the DrupalCI project is to have it dynamically spin up additional testbots on AWS on-demand, removing the need for human intervention to add additional capacity to the testing infrastructure when needed.