Provide performance information on testing bots

Posted by aclight on December 5, 2008 at 8:34pm

Issue Summary

I think it would be quite useful to provide additional information on the past performance of tested patches for each slave server. The information probably wouldn't need to be made publicly available on t.d.o. Useful information might include the average time it took to go through each step of the process (installing, checking out, running tests), as well as information on the outcome of tests run on the individual server.

I suspect that there is a lot of room for improvements in performance, but without some solid numbers it's difficult to see if any changes made have improved performance or not.

Comments

#1

Posted by Bart Jansens on December 6, 2008 at 11:01am

This would be quite useful, not only for benchmarking, but also to make test results available as fast as possible. If there are 4 idle testing clients, the master could pick the fastest one to run the tests.

From the logs, there appears to be quite a performance difference. The time to complete a full test run (from the recent t.d.o logs):

#4 : 7 minutes
#5 : 32 minutes
#7 : 17 minutes
#8 : 7 minutes
#9 : 25 minutes
#10 : 30 minutes

Of course, this information isn't entirely accurate because some patches include additional tests that could affect performance - but if we keep averages over the recent X number of runs, that should give quite a good idea of the performance. Don't simply count an average over all test runs, otherwise the older testing client could get punished when performance improvements are made to the testing code.

ps: i tried to avoid the word slave as in some parts of the world, people might get upset if we talk about making our slaves run faster and punishing them ;)