Automated performance testing for core

subscribe

For these purposes, log out will just mean to not send the session cookie.

If we want to simplify further, we can just chart HEAD performance over time and skip the compare against prior drupal. I'm not really sure how that works. We didn't have a toolbar or fields in D6. Do e test with a few fields on our nodes or not?

We might consider 6 months of refining the performance suite in contrib before we put it in core. Simpletest had years in contrib. But thats a detail. As catch says, anything is a big improvement over today.

For these purposes, log out will just mean to not send the session cookie.

If we want to simplify further, we can just chart HEAD performance over time and skip the compare against prior drupal. I'm not really sure how that works. We didn't have a toolbar or fields in D6. Do e test with a few fields on our nodes or not?

We might consider 6 months of refining the performance suite in contrib before we put it in core. Simpletest had years in contrib. But thats a detail. As catch says, anything is a big improvement over today.

Just subscribing for now. Don't have anything to add other than a big-ass +1. :)

Important yes but can't see how this blocks the release of Drupal 7.

I think that would be really awesome. I'm glad that performance is of importance in Drupal development.

Subscribing. Heck yeah! If possible, it would so rock to have this become part of the D8 dev process, so whenever a patch to D8 is submitted, there's a report as to how it impacts performance. Even if it's not fully integrated into the process until D9 development, it would still be awesome to be able to request a performance report for a particular patch. And if we're able to get anything along these lines up and running in some form within the next couple months, it will help with optimization efforts prior to D7 release.

Subscribing. I want to contribute to this if I can.

Thinking about this a bit...

1. The D6-D7 upgrade idea accomplishes two three things:
   1. pre-populates D7 database for testing in a relatively robust manner,
   2. provides a 'control' against which regression is measured,
   3. and ensures that the upgrade path doesn't break.
2. Since D6 is fairly stable performance-wise, it's probably not necessary to run the test suite on it every time, except to make snapshots. We could still compare D7 against the snapshots to detect regression for known bottlenecks.
3. Pre-populating the database with an upgrade seems a little "overkill" to me. All we have to do is basically create some nodes, terms, fields, menu items, what have you :) Our current test suites seem to do this pretty reliably. The only difference is that we'd be creating thousands of nodes, etc. to simulate the scale of an actual site.
4. With points 2 and 3, the upgrade idea is then only useful for ensuring upgrade path integrity. Seems like that could be done outside of the benchmarking suite, like in SimpleTest.
5. Benchmarking what we call 'known bottlenecks' is fine, but that doesn't work for every patch. Each patch or module may have some specific paths or API functions that they affect. I think it would be nice then, if there was a hook_benchmark_info() or a .benchmark meta file. That would let patches and modules define an arbitrary set of paths or API functions that would be run through the benchmark suite, making things less oriented around 'known pain points'.
6. I propose a sort of generic 'Performance rating' that the suite gives to a patch. While making it easy to read a benchmark at-a-glance, we would also need something like this to facilitate graphing each commit or week of commits. My ideas for this are:
   • Suite benchmarks HEAD and stores it as a snapshot, and snapshots are created with each commit. Benchmark returns a number of "points" corresponding to overall performance. I don't know how these points will be awarded :)
   • It could be that the benchmark does "assertions" like SimpleTest but instead of pass/fail, it registers points or percentages somehow.
   • When a commit happens. its benchmark results are compared with the previous snapshot(s), which would generate the rating. There could be two ratings, corresponding with D6 and HEAD comparisons.
   • I'm not sure what the actual rating would look like, but there would need to be an even score for no change, and an indicator when a patch totally flunks or when it awesomely improves performance by 5% or more.
7. A page on qa.drupal.org could automatically graph the snapshots, and show up-to-date performance improvements or regressions in HEAD. This would bring in an 'accountability' factor to performance testing in Drupal.
8. I agree with @moshe that this should at least start as contrib. Being in core is not required for PIFR integration either, since it can check out the module from CVS as it does for D6 SimpleTest, and apply patches to HEAD if needed.

I'm very excited about this project bringing more overall attention to performance, and accountability through documentation. Not to mention that we currently rely on poor catch for everything :)

1) Attempt and thoughts on update test #377856: Provide D6 to D7 update test
2) Assume we want to integrate this into qa.drupal.org in some manor? I am happy to help.

In IRC, catch made the point that the D6 benchmark is also intended as a baseline that the benchmark server/hardware is tested against to ensure a level playing field. I think it's smart to adjust for the test server's abilities, but I would rather do it with a one-time calibration than before every D7 benchmark. The calibration could also take into account filesystem, network, and CPU speed. The problem is, applying the adjustment correctly to benchmarks would be tricky and a flat multiplier would definitely not work. Ultimately, the benchmark results are relative to the test machine, so it might be that we just have to live with that variability.

In IRC we also discussed that apart from hitting common paths with ab/siege/jmeter, the benchmark suite should have support for what we call 'unit benchmarking' or 'microbenchmarking'. This would be basically, bootstrapping Drupal and running one function, say menu_get_item() or _menu_item_localize(), 1,000-1,000,000 times, averaging the run-time, and comparing changes that way. This way we can isolate the actual code from the bootstrap process, or if the change affects the bootstrap, isolate the bootstrap from the code.

Subscribing

This might be interesting
http://www.hell.org.ua/Docs/oreilly/webprog/pcook/ch08_26.htm

Hmmm register_tick_function() is really interesting. e.g.:

function tick_handler() {
  $GLOBALS['counter']++;
}

declare(ticks=1);
register_tick_function('tick_handler');
/**
 * Root directory of Drupal installation.
 */
define('DRUPAL_ROOT', getcwd());

require_once DRUPAL_ROOT . '/includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
menu_execute_active_handler();
unregister_tick_function('tick_handler');

echo "<br>Ticks:" . $GLOBALS['counter'];

Unfortunately not all statements are tickable. Typically, condition expressions and argument expressions are not tickable. Also body of internal functions aren't tickable. But we could write a code generator (we could use PGP module here) that creates a compiled version of drupal that increments the tick counter manually.

$a = 50;
$b = ($a >= 50) ? FALSE : TRUE;

Would become for example

$a = 50;
$b = ($a >= 50 && $GLOBALS['counter']++) ? FALSE : TRUE;

Just a crazy idea...

But isn't there some library that counts clockcycles or something alike instead of execution time? That would make comparisons a lot easier.

subscribe

I wrote this proof of concept one year ago:

http://drupal.org/project/scenario

The idea is to use the internal browser of simpletest to build a set of test scenarios. The runner can definitely be improved, but the idea is simple and sound.

@#18: Link no longer operational. Please see here alternatively:
http://0-0.at/doc/webprog/pcook/ch08_26.htm

A simple xdebug profiler log to qa.drupal.org would be nice - we could catch future performance bottlenecks.

@alippai: that would be a *huge file*.

subscribe

.

I second #11 wholeheartedly. Generalizing this would be great for d.o dev, but also testing of newly developed features offline.

JMeter testing is very hard for new users to configure, so an easy interface within Drupal would be a huge win. The difficulty would then be in parallel testing similar to JMeter. Is the end goal to be able to accurately simulate a load event (a la JMeter) or simply to measure individual request scenarios?

New to Drupal, first-time commenter, old-school web developer (Oracle).

I found my way to this issue after tracking #1064212: Page caching performance has regressed by 30-40% and reading http://ca.tchpole.net/node/2 and http://drupal.org/node/1020494.

At my local Drupal users group meeting last week, Drupal 7 was rumored, once again, as being slower than D6, at which point several attendees acknowledged that they'll take another look in a year or two. I'm one of several prospective Drupal developers in our city who is trying to decide between D6 or D7 as a starting platform, and when I hear these concerns, it gives me pause, even though I have a one-year runway for my first large project.

In addition, some of the new underpinnings of D7 are architectural in nature, but can appear esoteric to new developers. I myself am excited about the future possibilities with RDF and the database abstraction layer going forward, but most developers can't easily map those benefits to their clients' needs in the near to mid-term. D6 developers already know how to get around D6's issues, so they are looking for a compelling reason to take time away from their D6 projects/skills and invest in D7. Performance degradation is a fast way to terminate the conversation.

At my day job, for a semiconductor company, I interact with people who's sole job is performance analysis, where numbers in the nanosecond and picosecond range are bandied about as they compare our designs to the competition. These aren't the same people who design or implement the functional characteristics of a chip. They aren't emotionally invested in a particularly elegant architecture or an implementation of that architecture. They just tell it like it is. They spend all of their time writing and evaluating performance tests using a simulator before a design is sent to a fab, because it's a very expensive proposition ($millions) to implement performance improvements after the fact. It could require a complete respin of design, verification, layout and tape-out to the fabs. Fortunately, software is a bit more malleable, but some aspects still carry over. Once a product has a reputation, deservedly or not, for being slow, the damage can be hard to repair until the next product generation when people are willing to take another look.

If there was an automated performance test suite that ran on a periodic basis, either nightly or weekly, or perhaps on-demand, performance improvements and regressions could be identified fairly quickly and it would expedite the task of determining root cause. Perhaps an effort analogous to SimpleTest could be made a priority. It appears that serious thought has already gone into this issue and some of the tooling may already be at hand, both for core and contrib.

If my battle scars could talk, they would urge you to put the creation of a seamless and automatic performance testing environment at the head of the list, before making things potentially worse with enhancements or bug fixes for D8/D7.

Subscribe

Subscribe

I've been thinking about this issue for a while now and I've got a few thoughts for moving forward. I think it might be smarter to approach this from a code analysis standpoint rather than time-based benchmarking, per se. The trouble with benchmarking is that it is so exposed to external influences - other processes running on the machine, faster or slower machines, etc. Obviously you would standardize the machine and reduce those effects as much as possible, but that's tough, and somewhat difficult to sustain. Not impossible, mind.

What we're trying to establish is the degree to which an individual patch would affect Drupal's overall performance. We're often looking at a small bit of code - take #1135950: Remove static caching in t() as an example - and extrapolating what impact it is going to have. These patches might range from a typo fix in a comment, which would have no discernible impact, to something like a modification of the t() function, which might run 300 times per pageload.

So I'm curious about what tools are out there for analyzing code executions, less from a CPU time point-of-view than statistics on number of function calls, memory fetches, recursions, etc - anything that would be useful in predicting performance issues. A reasonable first step for automated testing, I think, is to raise red flags so that the patch can be scrutinized further, should possible performance issues result from a patch. So, looking to see if functions are called significantly more times, or significantly more memory is used, or if database queries are taking significantly more time to execute. Thoughts on this approach?

Great news! Now we have a beginning of performance metrics, what maybe we might need to add next after #3352851 is special weekly performance test that runs weekly or daily , part of the test would be creating 50,000 nodes, 5000 menu links, 500 taxonomy terms and 300 blocks, other entity types also, run through some operational tasks then monitor whether or not the performance improves or degrades between a week or a days worth of commits. Provide red flag on all the related commits between the tests period that slow performance down.

try to simulate some sort of realistic situation where performance would come into play.

There's only 2 child issues left, both are features so the meta could be considered fixed

Awesome work on this! Very important milestone!

Fantastic work, catch!