Determine the cause of inconsistencies in the test results between testbot machines

Four patches posted when http://drupal.org/cvs?commit=463094 was last commit.

Using the following script to roughly compare the results.

  header('Content-Type: text/plain');
  $ids = array(113699, 113704, 113709, 113714);
  $data = array();
  foreach ($ids as $id) {
    $b = new Browser();
    $b->get('http://qa.drupal.org/pifr/test/' . $id);
    $rows = $b->getPage()->xpath('//div[contains(@class, "tabs-tabset")]//table[contains(@class, "sticky-enabled")]/tbody/tr');
    $list = array();
    foreach ($rows as $row) {
      $list[(string) $row->td[0]] = (string) $row->td[1];
    }
    $data[] = $list;
  }

  print_r(array_diff($data[0], $data[1], $data[2], $data[3]));

I get:

    [Cancel account (UserCancelTestCase) [User]] => 293
    [Config settings form (SearchConfigSettingsForm) [Search]] => 178
    [Filter administration functionality (FilterAdminTestCase) [Filter]] => 218
    [Filter format access (FilterFormatAccessTestCase) [Filter]] => 262
    [Search number matching (SearchNumberMatchingTestCase) [Search]] => 321
    [Taxonomy upgrade path (UpgradePathTaxonomyTestCase) [Upgrade path]] => 1273

Just glancing at SearchNumberMatchingTestCase between the 4 patches I see that 2 are the same and the other 2 differ for a total of 3 values.

Running SearchNumberMatchingTestCase locally (through the interface and CLI) I get the same result four times in a row:

292 passes, 0 fails, and 0 exceptions

This would seem to confirm there is a problem. Interestingly the numbers the bot gets are all higher: 321, 324, 324, 329.