Tune SimpleTest

Working on getting some numbers.

After running first 10 test cases I get about 52% of the time spent installing Drupal. (54 seconds of 103 total)

This is the very basic code I used to benchmark he testing suite. Using .txt extension as there is no need for t.d.o to test this patch.

You will need to create a table called 'benchmark' with fields: type (int), elapse (float).

The results will be displayed after tests finish.

I already investigated this a while back (patch attached). We already can get a significant performance improvement by doing intra-test-class caching (ie. we don't reinstall Drupal between test methods, simply copy back the clean environment).

Caching test environments *between* test-classes (ie. caching a clean installation of a given set of modules) would require locking somehow (because tests can be run concurrently), and I'm not sure how to do this now.

*But:* copying DB tables in a clean, portable manner is not easy. For MySQL, we need to add NO_AUTO_VALUE_ON_ZERO... but this seems to break some part of Drupal (especially where we do INSERT INTO table VALUES (DEFAULT)).

As a conclusion: Crell talked about implementing a clean interface for copying tables in our database layer. We should bother him :)

would using mysql transactions help?
begin transaction
run test
rollback transaction
its a strategy i used in a previous life w/postgres

Possibly and then reseting the files directory, but that is easy and should remain the way it is.

Do we have proper access to transactions in Drupal 7 core?

Transactions can be installed anytime by converting tables to InnoDB. As for InnoDB in core see #301362: Default to InnoDB in MySQL. But shouldn't tests.drupal.org test installing Drupal core from scratch at least once a week?

#4 gives me: Notice: Undefined index: test_id in _simpletest_batch_finished() (line 403 of /home/catch/www/7/modules/simpletest/simpletest.module).

#7: much more often than once a week. It will test it on every patch as they are posted and the whole set of active ones once daily or every other day...

That is why tuning is important.

And to really test the install process it needs to be done on all the supported DB; currently MySql and PostgreSQL. But MySQL has different engines and I would suggest testing on MyISAM and InnoDB as well.

@catch#8: that's one of the issue with that patch, linked to the NO_AUTO_VALUE_ON_ZERO setting.

When assigning a new test_id, simpletest.module does:

This translates to:

And that fails when NO_AUTO_VALUE_ON_ZERO is set (DEFAULT translates to 0 so test_id = 0 on the first run, and a duplicate error is generated on the second run).

I still need to study how to solve this issue in a cross-engine manner (inserting NULL would work on MySQL, but probably fail on PostgreSQL).

Another thought, in order to support #10 comments more than one DB server on different ports with different configurations could help in supporting the multiple MySQL engine testing.

If the DB has an auto index, the value shouldn't be specified at all to use the feature. Specifying the value for the index of course will use the value given regardless of the DB engine. I can't find a similar setting as NO_AUTO_VALUE_ON_ZERO for PostgreSQL.

Specifying the value for the index of course will use the value given regardless of the DB engine.

Not at all :)

On MySQL, you are supposed to insert NULL (or omit) a column to have the autoincrement value:

NO_AUTO_VALUE_ON_ZERO affects handling of AUTO_INCREMENT columns. Normally, you generate the next sequence number for the column by inserting either NULL or 0 into it. NO_AUTO_VALUE_ON_ZERO suppresses this behavior for 0 so that only NULL generates the next sequence number.

But on PostgreSQL, serials are implemented by (1) sequences and (2) variable default values:

CREATE TABLE tablename (
colname SERIAL
);

is equivalent to specifying:

CREATE SEQUENCE tablename_colname_seq;
CREATE TABLE tablename (
colname integer NOT NULL DEFAULT nextval('tablename_colname_seq')
);
ALTER SEQUENCE tablename_colname_seq OWNED BY tablename.colname;

So you are supposed to insert DEFAULT (or omit) to get the automatic value.

Adding support for testing in separate environment is a totally different issue that I am looking into conquering later.

subscribe. very cool stuff.

Here is patch that caches install for entire test session. (base install without any non-default modules)

Then copies db over and installs test specific modules.

Note: I have not run entire suite, but initial results show 30+% speed increase.

This also needs cleanup as I do old style INSERT/UPDATE statements....

There is an error when running aggregator tests...bed time.

It needs to uninstall schema for custom (additional test specific modules) and leave all the default base tables.

I wish there was a way to quadruple-subscribe to an issue.

I want this in core so bad it hurts. ;) Will try and take a look tomorrow when I am more conscious.

Subscribe. It takes more than half an hour to run through all tests on my server, unless the server gets a hiccup halfway in and stops the testing.

Fixed the main concern I had...aggregator tests now run, but get a page title fail on one of them...

Interesting. (Subscribe.)

Coming back to this issue after a bit, I decided to write down the flow of the code so I could more easily visualize it.

I noticed an issue....hook_schema_alter()...so i can either see if additional modules have that implemented and only drop the test database then...or always drop the test database to ensure fresh schema.

The main benefit of not dropping will be much larger performance gain and considering hook_schema_alter() is one of the less used hooks...might be worth tyring.

sudo-code. Could move the create of base database to simpletest.module start of batch...but I'm think it will require code duplication...will look into.

In order to allow this to work will multi-process execution as provided by run-tests.sh I will need some sort of data structure to keep track of what test database are open for each base...

variable_get('simpletest_environment_prefixes');

This is not functional...but represents where I got tonight.

The getInfo() function per test case should be static...and is part of what is breaking this.

Feel free to take a look...I haven't even gotten it all to run due to the getInfo() issue.

Otherwise all the code is complete...who knows what it will do.

Blocked by: #376129: Change getInfo() to a static method.

After lots and lots of discussion #376129: Change getInfo() to a static method finally was decided on and made it in!

I'm am working on this.

I have made great progress! I spend a fair amount of time drawing out what I was trying to do and came up with a much better approach with an added benefit. The code currently runs in index.php so I can easily debug it, but I have it to the point where it will consistently re-unse environments and such.

You can see here that it reuses an environment between two test methods.

I have attached a patch of the essentials to what I am working on. NOTE: some parts need a fair amount of cleanup. (this patch isn't ready for testing)

Actually runs a test! Not how I want it to end up, but it works! (I might split off the second part into separate patch if it ends up being splittable)

Bed time now.

Needs #391340: Job queue API to work as I have planned and re-use prefixed test databases.

Subscribing

@boombatower: be careful, that's already a big patch, and it risks facing the same death as the previous attempt at refactoring Simpletest.

I suggest working on "baby"-steps:

1. first, we should refactor database prefixes to make them per-connection. The patch for this is nearly ready in #195416: Table prefixes should be per database connection, but desperately crying for attention. This way, you could have a "child" connection separated from the "parent" connection.
2. second, we should implement a DatabaseConnection::duplicate() method (or a similar name), that would copy all schema and data of an existing database. This needs to be done per database engine, because some engines might have nice shortcuts (for example on SQLite we could just copy the database file, etc.).
3. third, we need to add .test files to the registry, in order to be able to easily get the filename in which a test class is defined.
4. fourth, we need to implement a "test runner" that run an individual test. It would be perfect if this runner could avoid bootstrapping Drupal to BOOTSTRAP_FULL, in order to ensure that the environment is really clean for the ground up. For that we need at least 3 and probably 1.

There is really no need for a gigantic patch to do that ;)

#2: created: #449186: Implement DatabaseConnection::duplicate()

#4: This patch does that, it loads the test class itself in an environment which bootstraps in the prefixed database instead of trying to change to it and clearing statics.

Been meaning to work on #3 for a long time #449198: SimpleTest: Clean up test loading and related API.

Your right about splitting it up, much better approach...thx for bringing that up.

Note: the large patch you refer to that died...was very interdependent and required more work to split up then to just get on with life and commit...especially considering how much time I had spent on it...but oh well I let it die.