Deploy a PHPUnit system with a bottom-up approach

.

;

added NetBeans phpunit

Added UPal.

Adding tag.

Added Agile Unit Testing book page.

Duplicate of #1567500: [meta] Pave the way to replace the testing framework with PHPUnit and possibly rewrite Simpletest.

Can you add the Agile Unit Testing link to your issue ? As well as some (brief) description about problems and motivations ?

Also, please direct me to work in progress if any, Moshe told me to see with you directly.

typo.

Here is a very simple POC, adding the minimal required files to run the BlockTemplateSuggestionsUnitTest as a PHPUnit test instead of a SimpleTest one.

Usage:

Make sure you have PHPUnit 3.7 installed (via PEAR or composer).

cd my/drupal/install/root
phpunit

[31.10.2012 18:01:50 SYSTEM] Zend Monitor UI is disabled in CLI/CGI run...
PHPUnit 3.7.8 by Sebastian Bergmann.

Configuration read from D:\Profiles\sLecoy\Workspace\PHP\drupal-core\core\phpunit.xml

.

Time: 4 seconds, Memory: 9.25Mb

OK (1 test, 3 assertions)

I put it as Need Review for curiosity sake of the testbot.

Yes obviously, the PHPUnit_Framework_TestCase does not yet exists in the infrastructure :)

Just, if it is possible to have very early feedback on the approach I am taking it would be welcomed.

Corrected meta issue number.

Hi sun and thank you for the Platform sandbox access, although I can't see the branch yet... Maybe I have to creates it when I push?

About the classes:

• 2 plain phpunit classes: in the Drupal/Core/Testing package
• 3 legacy classes: in a Drupal/Core/Testing/Legacy package ?

About Guzzle, and PHPUnit, should I add an entry in the composer.json like core does ? Plus the actual libraries or this will bloat the patch size?

The top-level test suites declaration obviously does not work. We need a mechanism to discover and collect phpunit.xml files throughout the code base. (but omit files in any /vendor directory)

So you suggest having a phpunit.xml per module (e.g. system/phpunit.xml, user/phpunit.xml) then provides a mechanism for discovering and collecting them ?

I expect to be able to run tests for my (contributed) module from the top-level/root Drupal directory.

I don't understand what you mean by being able to run test from the top-level, you mean not from /core like it is proposed in the patch ?

OK for the general approach providing only the infrastructure. Secretly hope that it can happen for Drupal 8 :)

Some work in progress with utilities functions only... They are mostly a mix of UPal and SimpleTest 8.x.

Will start working on the branch aiming to implement setUp() and tearDown() methods.

Just created the branch in your sandbox sun.

Back on tracks I am working on Mink integration. Tests are fun to write and I am progessing pretty fast.

I am creating a 'testing' module which will hold the infrastructure along with the core/lib/Drupal/Core/Testing package/namespace. This testing module will contains - for now - integrations tests which will asserts the integration with PHPUnit for Unit tests, and Mink for functional ones. It will serves as an example for developers how to use the new testing system. Later it will replaces the simpletest framework. Also it will be possible to completely moves the testing module out of the core.

I have pushed a basic working test asserting the integration of Mink with Goutte driver.

I am working on database fixtures (http://www.phpunit.de/manual/current/en/database.html), and it introduces a concept of UnitTest, and DatabaseTest, they are differents.

Both can use the Goutte/Mink driver (I think at the rest module who can runs without the database), that's why I think to drop the name of WebTestCase. Instead of WebTestCase and UnitTestCase, there will be now DatabaseTestCase and UnitTestCase. If we want to add Web possibilities, we'll use a Mink Session through object composition instead of object inheritance.

I may requires some advices at this point on how to properly load a schema (per-class test) then load test data into it ? PHPUnit does a TRUNCATE but keep the schema between methods invocations for performances. Do we want to go this way ?

We could also load dump like we are doing in UpgradePathTest, but then again when ? On a per-test basis ? per-function basis ? What about setUp() and tearDown() ?

I am about to implement a functional test driven with PHPUnit and more specifically the Database test class, composed with the Goutte driver.

I am writing the new tests under the tests directory, thus a project will have the following:

- modules/
  - user/
    - lib/
      - Drupal/
        - user/
    - tests/
      - Drupal/
        - user/
          - Tests/
            - Fixtures/
            - UserCreateTest.php
            - ...
            - UserTestHelper.php

Its a good practice to separate tests from production code. Also we'll be able to make the transition more smoothly as it will make simpletest working as is, without having to modify the current code, and PHPUnit to look at the classes inside the 'tests/' directory. I am also implementing the sun's idea, replacing $this->drupalCreateUser() with $this->getHelper('user')->createUser().

I also strongly believe in "convention over configuration" that's why the example code assumes that tests classes will be under this directory. Also, the Helper classes will uses the following convention - for now - UserTestHelper.php.

As soon as i get the UserCreateTest functional test working with PHPUnit and Goutte, I'll provide a patch here for architectural review. In the meantime, you can check the branch on sun's Drupal sandbox where i'm committing work in progress.

Btw, thanks Berdir for pointing me out where to look :)

I have a few conceptual questions, in my last commit which aim to integrates a UserCreateTest, I have to set-up the database.

@see http://drupalcode.org/sandbox/sun/1255586.git/blob/refs/heads/test-phpun...

In the test class i'm using PDO as a sqlite memory implementation. So I am going to use this to set-up my fixture. I want also the module to be tested to implements its schema.

1. Can I inject this PDO object into the service locator (drupal_container) through the 'database' key as is ?
2. The PHPUnit philosophy is to first declare the schema, then fill it with test data. On each new test methods called, a TRUNCATE is applied on each tables of the schema. I'd like to keep this as close as possible to avoid new Drupalism. I need some help here to find a battle plan: Do we gather all modules needed by a class test then install their schema on setUpBeforeClass(), then let each methods adds their data ?

In #913086: Allow modules to provide default configuration for running tests, sun speaks about config, how does it relates to PHPUnit fixtures ?

Remember that I am taking two approaches in this issue:

1. A bottom-up approach: which aim to be as close as PHPUnit concepts and "standardized" which will break test suit compatibility (tests will need to be rewritten to take the new approach). I expect of these tests to be fast as the lightning speed.
2. A top-down approach: which will aim to provide a legacy layer, i.e. backward-compatible with Simpletest, and which will be a drop-in replacement for core tests.

I don't know if people are ok with the statement above so any advices/tips would be appreciated :-)

I just need directions/validations actually, a chat with sun, Moshe, or anyone involved in PHPUnit on IRC/mail should help me.

It seems that I can't inject a PDO object.

Recoverable fatal error: Argument 1 passed to Drupal\Core\KeyValueStore\KeyValueDatabaseFactory::__construct() must be an instance of Drupal\Core\Database\Connection, instance of PDO given in Drupal\Core\KeyValueStore\KeyValueDatabaseFactory->__construct().

So my next question is: Do/can we mock the DB object ?

Exciting work out there, I managed to completely remove the dependency upon drupal_bootstrap() and DrupalKernel. We are now on a super-minimal mocked environment.

By the way tests could not start if you were on a fresh copy of Drupal (i.e. without default/settings.php set-up). Now its working on a fresh install, in the minimal environment possible (I think).

Grab a copy of the last commit, and run command "phpunit" in the drupal/ root folder. There is 5 basic assertions, which will asserts your environment is mostly setup correctly with Mink, Guzzle, Goutte and PHPUnit + DBUnit.

Some code coverage and IDE integration to advertise...

added unit test remastered issue

Sure I'll have a look, I was not aware of this.

I am still on the PHPUnit part, then i'll work on the Database/Web test cases. I'd just like to have early feedback if possible because I know that working as submarine is never good. One day there is always a person who wakes up and says: this is not what we want to do :-(

I am committing really often and posting progress here, so i'd really appreciate to have early reviews as well.

EDIT: I simplified the process to be run with truely zero configuration: NO-PEAR, No-PHPUnit, No need to have a default site set-up (no settings.php required), etc. Everything is baked with the software, just do the following:

//In Drupal Root directory:
php phpunit.php
// Enjoy the speed.

Regards

EDIT: Updated issue:

It follows every agreements in #1567500: [meta] Pave the way to replace the testing framework with PHPUnit and possibly rewrite Simpletest except one:

• Install Drupal with (stripped down) testing profile + class-specific module(s).

When running UnitTest, we are not installing Drupal at all. For now the bootstrap only instantiate the class loader, and creates an empty container. The Kernel is neither created as well.

Added work in progress.

I encourage people following this issue to check this commit: http://drupalcode.org/sandbox/sun/1255586.git/commitdiff/711465fcd956cbc... and the one directly following.

I have factorized the code in builder object, which aims to manipulates MockObjects and provides default mocked services such as keyvalue, config.factory and then system.module, system.theme, etc.

The client use of this is the DatabaseIntegrationTest where you mock an environment very quickly. Of course you still have the full flexibility of MockBuilder since these classes extends the PHPUnit_Framework_MockObject_MockBuilder class.

Feedback appreciated !

Cheers and happy new year :)

I recently pushed the DatabaseIntegrationTest which asserts that installSchema() method works correctly with a mocked Connection object.

The principles are:

We creates a SQLite Connection object - here :sqlite::memory: - which is injected into the dependency container. This mocked connection will be then used by Drupal. We then install a particular schema from a defined module. Because I found module_list(), module_enable() was too dependant of many sub-systems (config.factory, keyvalue store, etc.) I decided to not use these functions in DatabaseTests. @see DatabaseTestBase::installSchema().

If we want to rely on module_list() and module_enable() like it is done in DrupalUnitTestCase, I think these belong to functional tests and not databases tests so I may re-introduce a WebTestBase class which performs a full-bootstrap on a minimal environment of pre-installed modules like it is done in UPal and with SimpleTests. These tests wont be as fast as UnitTests and DatabaseTests but functional tests are useful to test the interaction of systems.

The converting process should be more or less like this in the final state:

1. Unit tests will be converted verbatim, requiring some minor adjustment due to the difference of APIs.
2. WebTestBase tests which only asserts Database interaction - such as UserSaveTest - and not at all browsers capabilities, will be converted to DatabaseTestCase, requiring some refactoring.
3. WebTestBase tests which require browsers capabilities, focus on database calls and modules interactions will require more important refactoring, and a special handling at how to isolate them. Currently PHPUnit assumes database schema already exists meaning we have to write procedures to drop schemas between tests otherwise installation of a module will remain during the test suit.

The fact that PHPUnit assumes that the database schema already exists is a strenght when it comes to performance, but a flaw when we will convert our existing test suit. Althought I don't think it is a design flaw, for instance upgrade tests would be run on a separate database setup, we have to take this difference into account when writing tests because the level of isolation between tests will not be the same. I think however that it is perfectly acceptable if we know the consequences.

In this commit I partially converted the UserSaveTest (no-web based, only DB) to PHPUnit.

I had to mock the entity system, plugin system, and cache system as well. The test is verbose but it is a first attempt as a bottom-up approach to understand how to isolates properly systems.

I created a bunch of MockBuilder object for every subsystems such as ConfigFactory, Config, CacheFactory, etc. and fluid-style interface to configure them. Feedback is still welcome. Grab the last commit and run the test suit to see the UserSaveTest in action; the class is located at /core/modules/user/tests/Drupal/user/Tests/UserSaveTest.php.

Added code to fetch the project.

Hello jhedstrom, I greatly appreciate your feedback, thanks :)

I totally agree that the remote_addr should be parametrized in the phpunit.xml file, good point. Thanks for the bug fix as well, i'll push it.

It is wierd that you didn't found any test, are you using the HEAD ? I'll have a look on this.

Sylvain

Yes actually go to phpunit.xml and uncomment the following line:

  <testsuite name="Testing">
    <directory>core/modules/testing/tests/Drupal/testing/Tests</directory>
  </testsuite>

You can comment the other lines as they are not in sync with Drupal HEAD anymore (I get a Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException: You have requested a non-existent service "module_handler".)

The test suit Testing makes basic assertion about your setup. You can have a look at the User test suit which makes databases and more advanced assertions, and Core test suit which mock the filesystem and makes integration tests.

Alternatively, you can grab my last commit and execute the test suit.

Hey franskuipers thanks for giving me some attention, I guess we have some basic integration in #1901670: Start using PHPUnit for unit tests and you can focus on reviewing some mocking objects that I've proposed in this branch.

I think there will be some material to take and I'd be happy to continue working on it and/or to help on the subject. Let me know :)

Added plan