Benchmarking Drupal code

When attempting to create a large change in Drupal core, or even during the course of developing Drupal sites for clients, often you are asked to provide benchmark results to ensure that your change doesn't negatively impact performance (or to see how much it improves performance). This document will talk about the various strategies involved in benchmarking Drupal code.

Tools of the trade

• Apache Bench (ab)
• SIEGE
• Other performance testing tools

This document will discuss the ab tool in depth. Its a free, open source performance testing solution that comes with Apache and therefore is the tool that is most commonly used in the community.

In addition, you'll also need all the "normal" developer tools, such as:

• CVS (handbook page)
• Patch (handbook page)
• curl or wget for retrieving remote files (optional, but useful)
• A web server environment with PHP and MySQL/PostgreSQL installed. If the thought of installing these yourself gives you the heebie-jeebies, or you have an existing install you don't want to mess with, there are "all in one" packages that are as simple as unzipping/installing and you're up and running:
  • XAMPP (Windows, Linux, Mac)
  • MAMP (Mac)

Setting up a performance testing environment

Benchmarking is best done on your local computer or in a dedicated testing environment so that you can eliminate the possibility of network traffic and other factors that might bias the results.

Begin by navigating to your local web directory:

Perform a CVS checkout of Drupal:

This will retrieve a copy of the current HEAD version of Drupal into a folder called "head". You may also want a copy of the current "stable" version of Drupal for benchmarking purposes:

Install Drupal in the normal way and create the first user.

Following Dries's recommendations, we're going to setup an environment with:

• 2,000 users
• 5,000 nodes
• 5,000 path aliases
• 10,000 comments
• 250 terms
• 15 vocabularies

Download a copy of Devel module to get its handy devel_generate.module. Note that since HEAD can be volatile, always check the issue queue for updated patches.

Enable a few different node type modules (blog, page, story, forum, etc.) and then browse your site to /admin and use the 'Generate content' links to generate the sample data outlined above.

Notes

• You should disable MySQL query caching. Run this query:
  SET GLOBAL query_cache_size = 0;
• You should also disable PHP debugger plugins.

Benchmarking code

Begin by performing a

in order to ensure your source tree is up-to-date.

Then, execute the following from the command line before applying your new patch. This will establish a baseline from which to judge performance improvements or decreases.

The -c command specifies the number of concurrent requests and -n specifies the total number of page requests. In 99 out of 100 cases it is not necessary to specify a higher value than 1 for the concurrency. Increasing c will only put more stress on your server. This is useful if you want a general stress test, but it will falsify your results if you only want to test the impact of a particular patch on a particular Drupal page.

This will produce output similar to:

There are two important parts of this result:

1) Requests per second.

This is a good value to judge the overall performance impact. Higher is better.

2) The Connection Times table

The first row of the table specifies the headers for the columns. We are mostly interested in the second (mean) and third ([+/-sd]) column of the last row (Total). The mean or average tells us how long a page request took on average (in milliseconds). Here, lower is better. The value is actually not very different from the requests per second value; it's just that the computation is done "the other way around".

What makes this value interesting is the fact that ab provides the standard deviation of the value in the third column. The standard deviation is a means to judge how accurate a particular test result is. If the standard deviation is large, then this is an indicator that something wasn't working as it should, for example, the webserver was under high load or there was a network lag. If this is the case, you will usually see that in the table at the bottom (percentage of the requests served within a certain time) the last value is much larger than the first one. In this case you should investigate your experimental setup and try to reduce the standard deviation.

Now, apply your patch. Here we're showing that you can retrieve the patch file with curl and apply it with patch

Now re-execute the command above, and compare the results:

Interpreting results

Ideally, we now have two sets of values from our tests: The mean of the time taken per request and its standard deviation.

Without patch:

With patch:

We can see that the second value is higher by 12ms and we might draw the conclusion that the patch makes Drupal slower. However, there is also the standard deviation (which is rather large at almost 10% of the result). If you add the standard deviation of the first result and subtract the standard deviation from the second you will get:

Without patch:

With patch:

The order is now reversed.

This means the two results are within their respective standard deviations. This means that the test wasn't able to deliver a decisive result. In such a case you might try to increase the number of requests that ab does.

Ideally, your results should be several standard deviations apart to allow for a clear statement on the performance impact of a patch.

Drupal benchmark results

• Drupal webserver configurations compared
• Drupal vs. Joomla: Performance
• Drupal's database interaction