By Thox on
Following Dries' suggestion to run some benchmarks comparing Drupal's performance after an upgrade to 4.7 (CVS HEAD), I've gone ahead and collected some results.
I used general purpose testing machine with Apache 2.0.53, PHP 5.0.5, MySQL 5.0.15, and Zend Optimizer 2.5.7. The installation has 50 nodes, and 500 comments generated by devel.module.
The following values are in milliseconds.
Path: / (-9.4%)
min mean[+/-sd] median max
4.6: 200 1142 518.2 981 4176
4.7: 380 1034 208.4 1021 1802Path: /node/1 (-8.4%)
min mean[+/-sd] median max
4.6: 210 1155 530.5 991 3755
4.7: 220 1058 313.4 1051 2473Path: /tracker (-3.9%)
min mean[+/-sd] median max
4.6: 200 1084 337.6 1011 2313
4.7: 230 1042 275.8 1031 1922Path: /admin (+7.7%)
min mean[+/-sd] median max
4.6: 220 1306 638.6 1071 4566
4.7: 210 1419 602.8 1261 4035Path: /admin/modules (+16.4%)
min mean[+/-sd] median max
4.6: 190 968 244.8 951 1932
4.7: 250 1158 379.0 1071 2493[edit. I attached the raw logs. sorry, their order is not the best, despite I uploaded them in order. Also, they are from a previous run, smaller amount of content. chx]
| Attachment | Size |
|---|---|
| admin-modules46.txt | 1.31 KB |
| admin-modules47.txt | 1.31 KB |
| admin46.txt | 1.3 KB |
| admin47.txt | 1.3 KB |
| forum46.txt | 1.3 KB |
| forum47.txt | 1.3 KB |
| front46.txt | 1.29 KB |
| front47.txt | 1.29 KB |
| node46.txt | 1.34 KB |
| node47.txt | 1.3 KB |
| tracker46.txt | 1.3 KB |
| tracker47.txt | 1.3 KB |
Comments
Form API
admin/modules is slower, without a doubt, because of form API. As most web applications are 'read heavy' we thought this acceptable.
--
Read my developer blog on Drupal4hu.
--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.
node/add/story
Here's some results for
/node/add/story(3 runs for each version).Note: these results are using ab -c1, while the previous results all used -c5 (5 concurrent requests).
Is it worrying that the max is consistantly higher in 4.7?
I think for scalability
I think for scalability test, we need way more than 50 nodes. Is it possible to start with 1000 nodes and than scale it up to 10,000 nodes? It is fine for relative comparison of different version with 50 nodes, but I would still prefer at least 500 nodes for relative comparisons.
The results indicate that the variability of 4.7 is extremely high.
I think it is extrmely important to reduce the variability first before trying to find out the how to shift the mean.
Unfortunately, those result
Unfortunately, those result don't tell us much. The reason for this is the that the standard deviation is rather high. As an executive summary, the standard deviation needs to be smaller than any difference you are trying to spot. So if you have one result that is 1100 +- 300 ms and one that is 1200 +- 300 ms you should not state that there is a conclusion that the second test takes more time than the first. The standard deviation gives the probability that 68% of the results of this test fall into the given intervall.
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Slower
Still, if you look at the total time it took to execute the tests, you'll see that Drupal 4.7 is about 15%-20% slower than Drupal 4.6.
I'd like to see this investigated further because it is a significant regression. (This _might_ be due to the absence of URL aliases in the tests. Disabling the path module could make a difference.)
I agree that more testing
I agree that more testing needs to be done. Ideally on machines that are configured as webservers to rule out complications through other programms/configurations.
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Gerhard, if you want to have
Gerhard, if you want to have a shared hosting server todo some benchmarking, I'm very happy to share you my own account..
If you are interested, let me know by e-mail or in IRC..
While Drupal is now 15-20%
While Drupal is now 15-20% slower than 4.6, maybe it's worth rethinking about ch's split patch which brings us a 40% faster drupal. (chx please correct me when I'm wrong.)
Which will probably result in 15-20% speed up of drupal..
I think it's really worth rethinking, but hey.. That me offcourse!
PS: If you guys want to do benchmarking between an "Split Mode Drupal 4.7" and "Default Drupal 4.7 HEAD" let me know.. You could surely use my webspace for that if you guys like to..
Steef
Difference and SD
Dear Mr.
Take care of not negate the difference between two populations because they overlap... Otherwise you could not state that male are taller than women... Thus these sentenses are wrong.
But you are right, the test needs to be documented to ensure that the difference is not induced by chance or methodology.
Sincerely.
I am confused. Not satisfied
I am confused. Not satisfied with David's tests I wanted to do my own (as I have done in the past). I created a database with 500 users, 5000 nodes, 50000 comments, and some vocabularies. I made a copy and updated the copy to cvs. I indexed both databases for searching. I made some test using ab and found that the upgraded Drupal install is much faster.
Can anybody re-run those tests? I can provide the databases on request.
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Do you have numbers to show us?
- Robert Douglass
-----
My sites: HornRoller.com, RobsHouse.net
I didn't want to publish my
I didn't want to publish my numbers as they were so very different from David's that I thought they were wrong. Turns out they aren't. The major difference between my and David's setup is that he did not use any url aliases where my test database had an alias for each node.
To summarize: If you use a lot of url aliases, then Drupal CVS is much faster. If you don't, then 4.6 is faster. The reason is an architectural change made early in the release cycle: Drupal 4.6 would load all url aliases in one go (regardless if they were needed) while CVS tries to get an alias for each link on a page, resulting in a lot of queries. Ideas for improvements are welcome.
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Cvbge's db_query() patch
Last weekend, I committed Cvbge's
db_query()patch. The patch changes the waydb_query()works in order to support blobs. It was a requirement to make Drupal work on PostgreSQL. The change adds some overhead and renders the previous benchmark results invalid.Further testing
Here's new testing from today's CVS of 4.6.3 vs HEAD. I only looked at the front page on a live shared hosting server. 500 nodes and 2000 comments (generated by devel.module), with all standard modules and blocks enabled:
I'm a little shocked the server runs so badly - much slower than an old local test machine used with my previous tests.
Odd
This time around, your tests seems to suggest that 4.7 is ~10% faster than 4.6 (but once again the standard deviation is too large). Odd.
Same as before
This is the same as before, where I also got ~10% faster with 4.7 on the front page. I haven't tested admin screens or complex forms on this site yet.
performance of multisite
Drupal's multisite features could be calculated too. I don't know how to do benchmarks.
Previously I took the load averages and recorderd them with munin. I am enabling this again for my 4.6 host, and will report back when (if ever :) ) I upgraded to 4.7. I would love others to do so too. Benchmakrs are usefull, but IMO real world stats are very badly needed too.
http://www.linpro.no/projects/munin/
Ber
---
if you dont like the choices being made for you, you should start making your own.
---
[Bèr Kessels | Drupal services www.webschuur.com]
More horrid (but good) results
Here's some disgusting results from a live shared hosting server. I actually use this server to run a couple of sites! I'm disgusted to say the least:
That's /node/add/story on this morning's 4.6.3 and 4.7 checkouts.
that sd is strange
2539 +- 1990.8 ? There is a four times difference between min and max, that's strange.
--
Read my developer blog on Drupal4hu.
--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.
MySQL?
Sorry to come to the party late, but I wonder about the DB in all this? Have you turned the slow query log on? What is going on with the DB? It is hard to imagine that some of these larger max latencies don't have something to do with the DB. (appologies if this is old/irrellevant).
MySQL and PHP 5?
Another thought - the vast majority of folks will be using PHP and MySQL vers 4.whatever. Would be nice to have some stats there as well.
An informal test, I noticed
An informal test, I noticed CVS improved 7-12% on average upgrading from PHP 5.0.5 to PHP 5.1.1, so definetly worth it there! I'd like to see these same benchmarks though against 5.1.1 too!
Benchmarking
Did you use the Apache "ab" benchmarking tool to perform these tests? If so, can you tell me what parameters you used? I assume something like this:
ab -c 5 -n 100 http://server/page.htmI am trying to do some performance tests of my own, so it would be great to be able to repeat your tests on my server to get a comparison.
So far I have done most of my testing using "siege" (see http://www.joedog.org/siege ) as follows:
siege -c 32 -i -t 2m -d 5 -f url_list.txtThis seems to be a more realistic test since it selects a randon URL from a supplied file for each request, rather than just hitting the same URL every time like "ab".
Any thoughts or suggestions?
The question is what you
The question is what you want to test.
If you ant to test if a given Drupal page renders faster after soem change of code, then ab (with -c 1) is just fine. If you want to stress test your server, then tools such as siege make more sense.
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Good point...
Good point. I have now posted my results here: http://drupal.org/node/43478