HowTo: Benchmark Drupal code

Note: this page is mostly a result of me Googling around and jotting down some notes as I find them; I need someone who actually knows about this stuff to look it over and ensure I actually know wtf I'm talking about. ;)

Introduction

When attempting to create a large change in Drupal core, or even during the course of developing Drupal sites for clients, often times you are asked to provide benchmark results to ensure that your change doesn't negatively impact performance (or to see how much it improves performance). This document will talk about the various strategies involved in benchmarking Drupal code.

Tools of the Trade

• Apache Bench (ab)
• SIEGE
• other performance testing tools

This document will discuss in-depth the ab tool, as it is most commonly used in the community due to its being a free, open source performance testing solution that comes with Apache.

In addition, you'll also need all the "normal" developer tools, such as:

• CVS (handbook page)
• patch (handbook page)
• (optional, but useful) curl or wget for retrieving remote files
• A web server environment with PHP and MySQL/PostgreSQL installed. If the thought of installing these yourself gives you the heebie-jeebies, or you have an existing install you don't want to mess with, there are "all in one" packages that are as simple as unzipping/installing and you're up and running:
  • XAMPP (Windows, Linux)
  • MAMP (Mac)

Setting up a performance testing environment

Benchmarking is best done on your local computer so you can eliminate the possibility of network traffic and other things impairing the results.

Begin by changing into your local web directory:

First, perform a CVS checkout of Drupal:

This will retrieve a copy of the current HEAD version of Drupal into a folder called "head." You may also want a copy of the current "stable" version of Drupal for benchmarking purposes:

Install Drupal in the normal way and create the first user.

Following Dries's recommendations, we're going to setup an environment with:

• 2,000 users
• 5,000 nodes
• 5,000 path aliases
• 10,000 comments
• 250 terms
• 15 vocabularies

Download a copy of Devel module for its handy devel_generate.module. Note that since HEAD can be volatile, always check the issue queue for updated patches and such.

Enable a few different node type modules (blog, page, story, forum, etc.) and then browse to /admin and see the 'generate content' links and friends. Use those forms to generate your items above.

Note: you should disable MySQL query caching. Rub this query:
SET GLOBAL query_cache_size = 0;

Note: you should disable PHP debugger plugins.

Benchmarking code

Begin by performing a

in order to ensure your source tree is up-to-date.

Then, execute the following from the command-line before applying your new patch. This will establish a baseline from which to judge performance improvements/decreases.

In this command, -c specifies the number of concurrent requests and -n specifies the total number of page requests. In 99 out of 100 cases it is not necessary to specify a higher value than 1 for the concurrency. Increasing c will only put more stress on your server. This is useful if you want a general stress test, but it will falsify your results if you only want to test the impact of a particular patch on a particular Drupal page.

This will produce output similar to:

There are two important parts of this result:

1) Requests per second.

This is a good value to judge the overall performance impact. Higher is better.

2) The Connection Times table

The first row of the table specifies the headers for the columns. We are mostly interested in the second (mean) and third ([+/-sd]) column of the last row (Total). The mean or average tells us how long a page request did take on average (in milliseconds). Here, lower is better. The value is actually not very different from the Requests per second value, just the computation is done "the other way around". What makes this value interesting is the fact that ab provides the standard deviation of the value in the third column. The standard deviation is a means to judge how accurate a particular test result is. If the standard deviation is large, then this is an indicator that something wasn't working as it should, for example the webserver was under high load or there was a network lag. If this is the case, you will usually see that in the table at the bottom (Percentage of the requests served within a certain time) the last value is much larger than the first one. In this case you should investigate your experimental setup and try to reduce the standard deviation.

Now, apply your patch:

...and re-execute the command above, and compare the results:

interpreting results

Ideally, we have now two sets of values from our tests: The mean of the time taken per request and its standard deviation.

Without patch

With patch

We can see that the second value is higher by 12ms and we might draw the conclusion that the patch makes Drupal slower. However, there is also the standard deviation (which is rather large at almost 10% of the result). If you add the standard deviation of the first result and subtract the standard deviation from the second you will get:

Without patch

With patch

The order is now reversed.

This means the two results are within their respective standard deviations. This means that the test wasn't able to deliver a decisive result. In such a case you should for example increase the number of requests that ab does.

Ideally, your results should be several standard deviations apart to allow for a clear statement on the performance impact of a patch.

Drupal benchmark results

• Drupal webserver configurations compared
• Drupal vs. Joomla: Performance
• Drupal's database interaction