GHOP #127: Benchmark 5.x vs. 6.x

I did some of that earlier in the summer, but it needs to be more up to date and more extensive.

Subscribing.

Progress report:
Today I setted up the page correctly, I now have all pages you wanted free to bench in D5.5. I also created a little shell script for benching, my only problem is that I can't get the stdout of ab into a file:
I tested
ab2 -c1 -n50 *page*>testfile
and it worked :)
but when I did
ab2 -c1 -n500 *page*>testfile
Only the Header (all before x requests completed) was dumped into the file, the rest went into the console. Because X isn't running i'm not able to get the results :( This is why i can't present you something here, do you know what i could do? I'll also talk to chx when i get him in IRC morning. This post shall also prevent you from thinking i'm inactive ;)
Corni

REMINDER: All bench results are for anonymous users!
Computer Specs:
CPU: Core2Duo 2,24Ghz
RAM: 4GB
rest not really important...
I benched without X, aka only htop, ab and apache+mysql running (+ standards like cron...)
what i've done: installed D5.5 properly and did all tasks the status report wanted me to show a green
page ;)
Then i've downloaded devel module and generated the amount of needed things Dries recommended i've enabled all standard modules (which are installed _and_ active by default) and placed them on the frontpage
I also added the forum module (where I added one container with one forum) and enabled the search module
after i runned cron that many times and I had ~3,200,000 rows in search_index
I gave the anonymous user access to all pages (in the previous benchmark I did this was my fault :()
After all preparations I disabled the devel modules to be able to enable aggressive caching (it looks like not being compatible to it...)
Before benchmarking I set the query_cache_size to 0 ;)
Now, the result of the benchmarks are interesting. In short:
D5 is faster than D6. Poor dev's :P
All benchmarks are attached, i'll not compare each with the other correspondending others in the text, but i'll write some details.
Without caching:
D5->D6 index:
the index needs from 265 till 272 ms/run.
It served 3,74requests/sec
in D6:
it takes from 291 till 321 ms time. 3,4requests/second
Other pages:
taxonomy-term:
faster than the index, but 5 is better
node with alots of comments:
general a bit slower than the index, but D5>D6 :P
Forums index:
Because the Forums index is near empty and the forums aren't populated, that one doesn't really count. It's much faster than the index, and the differences between D5/D6 are smaller because of the much faster serving. BUT D5>D6
search:
Here D6 is a bit faster, but in overall the time needed is spread over more counts than in D5. The requests/sec are better, anyway :)
Normal Caching
Here you see that normal caching is important People not using it are silly people. Just see the difference of ~3-4 requests/sec and 100 requests/sec :P
Here is D6, as you think slower, and when looking at the requests/sec (and multipling with 60*60 for one hour) you see what wins when you run a server with high load -.-
And for the search: D6 is slower here, but this isn't a real in-the-wild test, no one will have a site where 100 visitors will search in 5hours for the same tearm. Here D6 is winning, because it's there general faster, even when the bench tells us something other.
Aggresive Caching:
it's a bit faster, but this are static tests, so if you want to see the difference normal caching/aggresive caching on your server, i'd guess you are not able to put real load away from your server. D6 is, as expected, slower.
If you want to show me that D6 is faster (it will be hard to release a D6: hey guys, it's just slower than D5, but please, use it, especially the big sites...) i'll appreciate any benchmarks done with conditions like above.
For any Questions, use my Drupal.org Contact Form (CorniI) or IRC (Corni)
edit: forgot the Benchmarks, here they go. Over each ab run there is a line describing what this is (just a small echo in my script)

Cornil

Can you please summarize the benchmarks in a table (or just [pre])?
Columns would be D5 total time, D5 req/sec, D6 total time, D6 req/sec

Rows would be each test title.

This will make it much easier to read.

Hi,
I'll compile a table morning @ g.d.o, but could you review the task just without it or tell me how I can include the table @Google's Issue tracker?
edit: aclight said i shall just save the generated html and upload a .htm to Google, so forget this post ;)

A more digestable table view of the results would be great.

here it goes...
http://groups.drupal.org/node/8274

Thanks for the summaries, not now it is much more comprehensible.

Dries' benchmarks show a concurrency level of 20, so a concurrency level of 1 (= no concurrency) is not too realistic on a site where performance is important. Also overall times like 0.6 seconds taken for all tests in a benchmark are not really telling. Your machine might go through some hops in that second which drastically effect the results.

So please increase your concurrency as suggested in Dries' tests, and increase time taken for tests (by increasing number of requests with the same amount for both D5 and D6) to at least 1 or 2 minutes each. Make sure there is nothing transient running (eg. a nice scrollkeeper-up process sucking up CPU resources on a desktop machine :)

Thanks for taking this up!

Dries' benchmarks show a concurrency level of 20, so a concurrency level of 1 (= no concurrency) is not too realistic on a site where performance is important.

I'm sorry, but I think you're wrong. Setting concurrency high will test my computer's web server capabilities, not Drupal's speed. c=1 is just fine :) For the length of the benchmarkes: I know, it should be more, BUT I wanted to do all benchmarks @ -n500, but runned into strange ab2-bug, wghenever it wanted to display
x00 requests completed
it disconnected my output redirection into a file to capture the benchs, so I talked with chx about it, and he said that 500 is quite much and that 100 will do. But you can see from the Search result without caching, that there aren't any strange disruptions, and the other bench's without caching are between 20-30 sec's (without strange hops), so they are fine, too. And for the Desktop comment: X is quitted, and with it, all processes started isinde the X tasks ;)
Corni

I added and corrected the last entry on the groups.d.o page, please comment/review :)

Weird that the most performance is lost when caching is on isn't it?

I have also heard that the ab "n" value needs to be in the thousands in order for it to be valid.

It appears that there are some strange results (I reformatted the ab results above for easier reading). For example, I believe that this shows that some of these tests are not conclusive:

              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     6    8  28.3      6     289
Waiting:        4    8  27.6      6     282
Total:          6    8  28.3      6     289

I think this says that the mean is 8ms and the standard deviation is 28ms.

If you are having trouble capturing the output I suggest you run this via a remote terminal from another machine and then copy/paste the scrollback from the terminal. If you don't have an extra machine to login from remote then, I suggest you use screen - "man screen" and look for the option "C-a H" which will save a log of your screen session to a file.

And there is some good work done here, so updating status.

The test that says:

Complete requests:      100
Failed requests:        99
   (Connect: 0, Length: 99, Exceptions: 0)
Write errors:           0

Needs to be repeated without failure.

I agree with kbahey on this.

So, things to confirm in the results from the test to know that they are valid:

1) standard deviation is low (should be less than 2% of the mean)
2) 0 failed requests

We want to make sure that the next time Cornil runs this is the last time - anything else he should look for in the results to make sure they are valid?

This is good that someone has picked this up, but when I did some benchmarking several months ago (part 1, part 2) the biggest improvements BY FAR were for authenticated users. Anonymous results are great, but if we don't have those authenticated numbers it will be a shame. (since as I say, they are considerably higher percentage improvements than those for anonymous users)

I look if yI can do the benchmarks as fast as possible, but i'm away on the weekend :(
IMHO, it's very important to figure on anon user's, because if you want to present your cool new company, nobody needs to register to see your cool content.
Corni

So that everyone is aware of his schedule, the last time I talked to Corni he said that he hoped to finish this during the weekend of the 2-3rd.

@Corni - Drupal is used for many purposes, but one of the most common ones is for "community websites" where most users are logged in. There are large sites with +95% of visitors being logged in. One of the top requests from the "State of Drupal" Survey was improved performance for authenticated users - http://buytaert.net/files/state-of-drupal-september-2007.pdf

OK, I got some space in my schedule, D6 is done on -n200 (talked with chx about it) and sd is mostly below 2 :)
I hope I get time for D5 tomorrow , if yes, then the updated table etc will follow saturday/sunday
Corni
edit: D5 is done, too :)

I updated the table with the current results, I just miss the percentages, they will arive tomorrow together with an updated writeup. Here is the raw data:
see #25 for raw data

Cornil

Since the raw results are very big, it is better to create a .txt file and attach it, rather than post it in line, making it too big.

subscribe. thanks cornil.

REMINDER: All bench results are for anonymous users!
Computer Specs:
CPU: Core2Duo 2,24Ghz
RAM: 4GB
rest not really important...
I benched without X, aka ab and apache+mysql running (+ standards like cron...)
what i've done: installed D5.5 properly and did all tasks the status report wanted me to show a green
page ;)
Then i've downloaded devel module and generated the amount of needed things Dries recommended i've enabled all standard modules (which are installed _and_ active by default) and placed them on the frontpage
I also added the forum module (where I added one container with one forum) and enabled the search module
after i runned cron that many times and I had ~3,200,000 rows in search_index
I gave the anonymous user access to all pages (in the previous benchmark I did this was my fault :()
After all preparations I disabled the devel modules to be able to enable aggressive caching (it looks like not being compatible to it...)
Before benchmarking I set the query_cache_size to 0 ;)
Now, the result of the benchmarks are interesting. In short:
D5>D6. It is always faster for anon's except for serving uncached search results. This applies to the new benchmarks, too!
All benchmarks are attached in one txt, a nice table of them is under http://groups.drupal.org/node/8274
Also, I updated the status here, I need a review ;)

catch: isn't planned at the moment, but you could try to get another volunteer do this :P

I think this is long-since closed?

If you think it should be reopened, please put it into a component other than "documentation", which is for API documentation in Drupal core.

If it's something that needs work in the Handbook, it belongs in the "Documentation" project, rather than Drupal project, component documentation.