Based on a discussion after the server meltdown, the need for a benchmarking suite is now more urgent.

Since drupal.org is now moving to a dedicated cluster of sorts, having a way to measure performance on various configurations is more important than ever. For example, bottlenecks that are invisible on the dedicated environment may be serious enough on shared hosting.

My idea is as follows:

- Using some scripting, create a test database that is large enough for a medium site (1000 nodes, 2000 comments, 500 users). As we progress on this, we can create another large database (10s of thousands). It will only contain core modules (no contribs).

- Create instructions on how to load this database

- Create some Apache Benchmark scripts (ab) that go against this database.

- Create instructions on how to run ab against a drupal installation that has the above database loaded.

By running the above on different configuration (shared vs. dedicated, Linux vs. Windows vs. Mac, caching on vs. off, 4.6 vs. 4.7, ....etc.) at least comparative figures can be established. Coupled with info from page generation times, we can know where the bottleneck is.

As new releases of Drupal are created, the database will be updated via update.php, so it can go forward with releases.

The above is a rough outline and subject to a lot of refinements and discussions. Please comment below.

Comments

bertboerland’s picture

bertboerland commented
Title: Drupal Benchmarking Suite » take a look at

http://opensourcetesting.org/performance.php, i think most is there. I like jmeter

mind you, a fool with a tool is still a fool (no offense), so you need to define what you are measing and if you are really measering that (and not the client(s) for example).

kbahey’s picture

kbahey commented

(Hmm. I will never get used to this dry Euro-humour!)

Anyways ...

Of course, the test should run on two machines, otherwise the load created by ab itself would skew the results. The machines should be on the same LAN, otherwise we are measuring network lag as well.

I have several at home, and can run ab on one of them, and Apache/PHP/MySQL/Drupal on another.

The idea here is to build a database which anyway can use to benchmark anything they feel like benchmarking.

Examples are:

- You rewrite some module and wonder what the performance will be for 50 simultaneous users. You do a before test and after test and compare them.

- A new release of Drupal come out. What is the performance like compared to the previous release?

- You want to add module X from contrib, but you are unsure what the impact on performance will be. You do a baseline test without the module, and then install the module and see what the impact on performance is.

Of course, this approach is not an answer to everything (nothing is!), but as you can see, it provides a tool set for others to build on.

jmeter looks like a capable tool. More suggestions/comments welcome.

As a side comment: The screen pages with page generation times that Dries publishes every now and then. are they a standard code part, part of the dev module, or something else? If we find a way to log dev_timer somewhere for each page, with the URL and time taken, then an average for all the same URL. Perhaps logging this to a db table or a flat file. Or perhaps it is there already.

moshe weitzman’s picture

moshe weitzman commented

it is there already. see the links nuder admin/logs ... you have to enable stuff under admin/statistics

capmex’s picture

capmex commented

I guess this can be achieved independently through the work and effort of a small group. For the comments, I can see it will require third party software, a special infrastructure and people with at least basic knowledge on testing web applications.

Since we will only be testing drupal using third party software in a controlled environment, it has to be an independent drupal project and cannot be part of drupal development. Although in the end core developers will benefit from having such information.

kbahey’s picture

kbahey commented
Title: take a look at » Drupal benchmarking

It does not need to be in core, although it will test core mainly, as well as contrib modules, as needed by the person who runs the suite.

The third party software will be the client part, that is, the part that simulates a certain number of users browsing a Drupal web site.

ab is part of Apache, and fairly straightforward to run. Jmeter has been mentioned.

Regardless, what I am proposing is that the end result is two parts:

A. A test database, in the format of a MySQL dump (sorry PostgreSQL guys, I don't run that)
B. Instructions on how to load a Drupal system using a client (e.g. ab)

Someone else may clone B using jmeter, or something else. It is just a document on how to install, configure and run the test suite.

bertboerland’s picture

bertboerland commented

what you need:
standard drupal install (to test)
standard database
100 userid's or so
network
client

no make sure you login with /different/ userid's from the client, do a post, throttle the server etc. just doing an anonymous get like ab does, wont be representive since cache / postings will be different in real word. I can help if you like?

peterx’s picture

peterx commented

Some things of interest to me and my sites.

Testing the Opteron server and the Xeon servers side by side would be good. Something worth repeating as each layer of software is converted to 64 bit.

Testing with PHP cache on and the application style caches, Smarty et al, switched off, would be useful as the PHP level caches save server resources while the application style cashes are close to useless in modern dynamic web sites. Repeating the test across PHP 4 and PHP 5 would be better as it would help people plan the conversion.

Testing a load of logged on users against users not logged on would be useful. The logged on users should get lots of flexibility at the expense of performance while anonymous users can have their code optimised for the lowest overhead. Is there really much of a difference in Drupal?

The tests would also need to record of the use URL aliasing and other options so we can see if any of those options hit performance. There are community sites that would use all the options if they did not hit performance and would put up with a less capable site if leaving out an option helped them avoid a server upgrade.

Some other sites might have already run tests of overheads with specific modules and options turned on then off. Those outside tests would be of use if they were specific about the changes made and the resultant overheads.

magico’s picture

magico commented
Version: 4.6.0 » x.y.z

There were significant improvements in Drupal scalability in the last few months.
There were also several comments and posts about benchmarking and optimitization, including the "idea" of adding some features to the devel.module

So, is this issue being addressed and can be closed?

LAsan’s picture

LAsan commented
Version: x.y.z » 7.x-dev

Feature request go to cvs.

catch’s picture

catch
he/him
Primary language English
commented
Status: Active » Closed (fixed)

Closing this, been lots of benchmarking done.