[meta] Pave the way to replace the testing framework with PHPUnit and possibly rewrite Simpletest

atoum

I just get a small presentation of it. 30s after I was searching if someone was thinking to move away of us hacked Simpletest.

It's nice, easy, fast, modern and flexible.

I'm the creator and lead developer of atoum and i have no problem to help you to use it instead of SimpleTest.

Atoum is more attractive and more easy to use.

For a "web client" to do functional test http://mink.behat.org/index.html

mink is integrated out of the box with PHPUnit, Behat and Symfony2 but any integration is certainly easy http://mink.behat.org/phpunit.html

And for functional test in natural language, take a look at behat (it's used mainly for https://en.wikipedia.org/wiki/Behavior_Driven_Development ...).

EDIT: didn't see the issue summary evolve from the last time I read it.

Updated issue summary.

If you find this paper good Agile Unit Testing you can add it as a resource guidelines regarding unit testing.

Updated issue summary.

Added a POC of a block unit test re-factored to use PHPUnit #1801176: Deploy a PHPUnit system with a bottom-up approach.

FWIW, there are already actual sites using PHPUnit tests in a Drupal setup: some customers of mine wouldn't bend over their testing and CI practice and chose to develop unit tests for their custom code using PHPunit only, no Simpletest runner.

Added Unit test.

Added Issue #1801176 in the Unit test section.

typo

More about this:

• just as an experiment, I started adding PHPunit tests on object-based code in a recent project, and running it through Drush essentially means just having to make the tests inherit drush_unit_test_case and running drush_bootstrap at the beginning of each test (and cleaning every DB change at the end of the test. Mocks really help with object-based code, and sometimes just wrapping the basic core function call in an object method is all it takes to enable actual unit-testing via a mock
• regarding Behat, Mink + Guzzle + Goutte is a nice and fast way to run integration tests, but they completely miss the JS testing, while more and more of our code becomes JS-dependent. Something else is also needed to support it (Selenium2, Sahi, Zombie...)
• something I would very much like to see is PHPcov integration, which comes as a given with PHPunit and not (AFAIK) with our current simpletest: it would enable devs to see how far they are from C0 coverage when pushing code. While 100% C0 coverage is neither necessary nor sufficient, it is a useful direction
• we really should have something to enable newbie devs to run and write tests without extra complication: the whole topic of testing is intimidating enought as it is

How can we take this initiative further?

I was very happy to see PHPunit sneaking into core with #1901670: Start using PHPUnit for unit tests! And because I want the learn and get experience with PHPunit I started to work on this in #1938184: Convert Drupal\Core\Utility\Color\ColorTest to PHPUnit and #1938068: Convert UnitTestBase to PHPUnit.

Looking around more into all PHPunit related issues I realised it is not a concerted effort yet.

• Sylvain Lecoy has worked 4 months on #1801176: Deploy a PHPUnit system with a bottom-up approach with great passion in the Platform sandbox. How is this work bypassed by #1901670: Start using PHPUnit for unit tests? I can not talk for Sylvain, but I would be a bit frustrated.
• There are a number of issues for PHPunit in the queue.

Some ideas to bring this further:

• Can we investigate if and how we can integrate (parts?) of #1801176: Deploy a PHPUnit system with a bottom-up approach into core after #1901670: Start using PHPUnit for unit tests?
• Can we use the Platform sandbox to apply and test some of the issues in one branch
• It would be nice if we can ditch simpletest when we start D9 at the end of the year

I would like to help/carry on with this issue, but need some guidance as how to proceed.

The PHPUnit Blocker tag is moving a few parts of Core over to PHPUnit.

Added Issue #1872006 in the Unit test section.

This is 9.x territory now, alas

So I've started plugging on a bridge class LegacyWebTestBase that extends UnitTestCase (PHPUnit) and has 1:1 methods with SimpleTest WebTestBase
I've pushed this to a branch in my core sandbox
I have it to the point where you actually see the class make Curl requests (I have some scaffolding in for Mink/Goutte but leaving it with Curl/existing logic until stuff is at least running).
Commit summary (from sandbox)

ea9a7f6 Composer update, new vendor files
d7fb3dd Added php unit test listener
9f9f13e WIP on legacywebtestbase, up to 2175
c7014d4 [#die] Views fixes to make tests not need t()
afc8208 Changes to phpunit config
1ea406e Changes to LegacyWebTestBase
a3be798 Tour uses legacy test base
a3e9405 Disable verbose for now

To see where I'm at

cd core
./vendor/bin/phpunit

It will error out because no $_SERVER['TESTING_BASE_URL'] provided.
Edit your settings.php and add $_SERVER['TESTING_BASE_URL'] = 'http://d8.local'; using your local url.

./vendor/bin/phpunit

Sample output

PHPUnit 3.7.21 by Sebastian Bergmann.

Configuration read from /var/www/d82/core/phpunit.xml.dist

...............................................................  63 / 504 ( 12%)
............................................................... 126 / 504 ( 25%)
............................................................... 189 / 504 ( 37%)
............................................................... 252 / 504 ( 50%)
............................................................... 315 / 504 ( 62%)
............................................................... 378 / 504 ( 75%)
............................................................... 441 / 504 ( 87%)
..............................................................Starting 'testTourFunctionality' 
F 504 / 504 (100%)
Test 'testTourFunctionality' failed.
Failed to set field name to e8hxk751
Failed asserting that false is true.mkdir() [<a href='function.mkdir'>function.mkdir</a>]: Permission deniedchmod() [<a href='function.chmod'>function.chmod</a>]: Permission deniedfile_put_contents(/var/www/d82/sites/default/files/php/service_container/.htaccess) [<a href='function.file-put-contents'>function.file-put-contents</a>]: failed to open stream: Permission deniedmkdir() [<a href='function.mkdir'>function.mkdir</a>]: Permission deniedchmod() [<a href='function.chmod'>function.chmod</a>]: No such file or directoryfile_put_contents(/var/www/d82/sites/default/files/simpletest/797998/php/twig/.htaccess) [<a href='function.file-put-contents'>function.file-put-contents</a>]: failed to open stream: No such file or directoryVerbose output from http://d82.taco/user:http://vendor/bin/sites/default/files/simpletest/verbose/-0.html

Time: 30 seconds, Memory: 66.25Mb

There was 1 failure:

1) Drupal\tour\WebTests\TourTest::testTourFunctionality
Failed to set field name to e8hxk751
Failed asserting that false is true.

/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:735
/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:2680
/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:1845
/var/www/d82/core/modules/tour/webtests/Drupal/tour/WebTests/TourTest.php:36
/var/www/d82/core/tests/Drupal/Tests/LegacyWebTestBase.php:832

FAILURES!
Tests: 504, Assertions: 1010, Failures: 1.

Note This will hose your existing site, we still have leakage between the containers/compiled twig

Note This will hose your existing site, we still have leakage between the containers/compiled twig

I still really think we should ditch simpletest and rely solely on phpunit. We have tried to plug these holes for *years* and been unsuccessful. The mental burden will be so much lower if we only have one Drupal, and rely on phpunit to be our test runner. I will personally write the (small) patches to help test bot run phpunit instead of simpletest.

Thanks for working on this.

The test bot is already running PHPUnit and there's issues to convert existing core unit tests to use those. No bridge, just one-by-one rewriting the unit tests.

What we don't have is issues to convert functional tests away from PHPUnit - we're still going to have some functional tests, and PHPUnit isn't really designed for those. For that I think it'd make sense to start using Behat against core - which would get us functional JavaScript testing as well.

@catch I wasn't trying to bridge unit test, but functional tests aka WebTestBase.
The idea being get something going with PhpUnit as the runner and then integrate Mink as the browser instead of SimpleTest.

I am just starting to learn D8 and Symfony and was looking into how both handle functional testing. I found functional testing in Symfony very intuitive (http://symfony.com/doc/current/book/testing.html#functional-tests).

I started reading this issue and could not help but wonder why we wouldn't consider extending Symfony's WebTestCase and use it to replace SimpleTest. My guess is it might require Drupal 9 to include more components from Symfony, which seems to be the direction things are going for D9.

I am sure I am missing something ... but sometimes it is worth asking some simple questions when trying to solve big problems.

@jrockowitz - thats a great find! Symfony's webtestcase simulates an http client, but there is no actual http involved. thats a nice speedup. See http://symfony.com/doc/current/book/testing.html#working-with-the-test-c...

I'd be fine with Symfony's WebTestCase or Behat for our functional tests. Behat uses http which is unfortunate but writing tests is approachable to anyone.

Actually, you don't have to use HTTP at all to run behat, especially when using the Drupal extension: tests can run either using the API, using Drush, OR using and actual HTTP client (Blackbox API). It's simply a matter of tagging your scenarios so that the extension knows how to run them, and configuring the drivers in the test config.

http://www.dspeak.com/drupalextension/drivers.html

I wonder if making this 1:1 compatible phpunit+mink replacement for simpletest means this could be in 8.1.0 in a semantic-version world?

If I am reading it correctly, http://extensions.behat.org/symfony2/ provides a Behat driver that runs requests against the Symfony kernel directly - this would seem to give us the best of both worlds - write functional tests in Behat, but avoid the HTTP overhead.

Just a note, I'm not sure that Drupal's bootstrap is at the point yet that it could handle a non-true-HTTP functional test. A lot of progress has been made in that direction but especially things like the installer may not be able to handle it. I know Mark ran into lots of issues between simpletest and the installer when trying to separate out the bootstrap into better kernel layers.

That said, if we could pull off simulated-request-based testing that would a HUGE win, and I fully support it. Having taken that approach on a Silex project recently I much preferred it to what we do now in Simpletest.

Added issue 1901670.

simpletest is not broken beyond any repair. As I have mentioned in several issues: as long as you need to install a child Drupal to test and run it with Drupal you will have problems. That is irregardless whether we use our own browser or someone else's.

I am told Ditch is not a PHP project being considered. What do I know.

PHPUnit is already in.

If we're going to make any testing changes apart from individual conversions of some functional/integration tests to PHPUnit, I'd want to focus on js testing. JS testing will also individually convert some simpletests that are working around not being able to execute js, then there's a discussion to be had about what happens to the tests that don't belong to either browser testing or PHPUnit.

The issue for that is #237566: Automated JavaScript unit testing framework , so postponing this one.

Spoke to omissis in irc who's been experimenting with this.
Encouraged him to create a new issue to track his approach - he created #2283257: [meta] Porting Drupal Integration and Functional TestCases to PHPUnit
Have added it as a child.

BrowserTestBase is in

Everything here except actually removing simpletest can be done in a minor version.

I think this is now closed - fixed

Now that #2469721: Add functionality to store browser output to BrowserTestBase is in, BTB and JTB have feature parity.

Time to open a meta to start the conversions?

Have we marked everything in Simpletest @deprecated at this point?

No. We haven't deprecated WebTestBase yet. Maybe that is the next step?

I think before doing that, we need to ensure all the documentation for writing a test is updated to reflect the New World Order™ because by deprecating WebTestBase, we'd be in effect requiring all future web tests to be written this way.

Also, we should probably convert a bunch of existing tests to see if that (feature parity) is actually true. We thought that for the new KernelTestBase too and fond a whole bunch of issues and missing features will actually porting the tests.

That sounds right. Once we're getting there with the conversions, I think we could add the deprecation notice.

And then there is #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase).

I opened #2750461: Remove Simpletest UI because we don't want to maintain a graphical test runner which is almost a duplicate.

This issue was postponed in 2013 to get a javascript testing framework, this is now done with the introduction of JavascriptTestBase using PhantomJS.

Since this issue is old and contains all kinds of details that we already solved which are not relevant anymore I suggest to call this fixed in the sense that this successfully paved the way. Thank you all!