[Meta] Use Behat for validation testing

I do have projects using Behat, but at the "whole site" level, not individual modules.

We've been extending the DrupalContext provided by DrupalExtension, as well as extending the Behat Buzz-based WebApiContext https://github.com/Behat/CommonContexts/blob/master/Behat/CommonContexts...

FWIW, (a) Behat 3 is almost launched and changes things a bit, (b) Buzz, which supports the WebApiContext it not very much alive and mixing Mink-derived steps with WebApi-derived steps doesn work (c) The WebApi context itself uses old-style includes, and already needs some tweaking to work with current practice.

All in all, with Rest getting such a larger role in current practice, we should use steps similar from the WebApi ones, but based on the Mink.

Not sure where this can take us regarding the larger issue you describe.

We use Behat for several client projects at Tag1 as well, and also started a proof of concept for automated performance testing using it a while back.

Per https://github.com/Behat/Behat/issues/148 it looks like 3.0 supports running multiple suites at the same time, that would let us do $module/features with their own scenarios and context classes, as well as whatever site/profile level tests we'd want to ship with. Even without that we could collect tests outside behat and run them individually.

Two issues to open, but just putting notes here for now:

We should probably at least open an issue to look at porting DrupalExtension to 8.x/core. There's some things in there we probably can't rely on in core, like drush, but that's for the issue to sort out.

Also how to deal with the set up of the tested site, i.e. we'll need to have a clean install, but do we do that per scenario? per feature? Could we make it optional via a tag? Behat already has the events to handle this so it's mainly a choice on our side.

nod_ pointed me to this issue.

I'll note that this might lend an opportunity to remove the vendor directory from core. I would assume we would add drupalextenion, behat, mink, etc to require-dev. Granted, there is some overlap with core, but just adding composer.json below to an empty folder creates 12 mb of libraries, which might be fine, worth some discussion.

{
  "require-dev": {
    "drupal/drupal-extension": "*",
    "behat/mink-extension": "*",
    "behat/mink-selenium2-driver": "*"
  }
}

Updated #2229747: [d8] settings() has been removed and VERSION constant removed and was able to run a really simple behat test against 8 with no issues.

We should probably at least open an issue to look at porting DrupalExtension to 8.x/core. There's some things in there we probably can't rely on in core, like drush, but that's for the issue to sort out.

Drush isn't mandatory, just one of the available drivers for communicating with Drupal. The Drupal driver would be the way to go. Drush is used for the testing of the extension itself (see .travis.yml, and the bundled Behat features).

I'd be interested in discussing an approach where the entire extension isn't forked and placed into core, but subcontexts for individual modules are created and put into core.

Also how to deal with the set up of the tested site, i.e. we'll need to have a clean install, but do we do that per scenario? per feature? Could we make it optional via a tag? Behat already has the events to handle this so it's mainly a choice on our side.

I'd really like to not see a clean site per scenario. Even per-feature seems overkill for this level of testing. That isn't to say that tests should rely on data created from previous scenarios, but if a reasonable effort is made for scenarios to clean up after themselves (the Drupal Extension currently clears out any users, nodes, and terms created during a scenario), then we can avoid re-installing for every feature, saving a tremendous amount of time spend on test runs.

Had a lengthy discussion with nod_ about this - here are the main points

• At PreviousNext we chose the zombie driver over selenium/phantom because of speed concerns
• The Zombie driver works by creating a Nodejs server (a .js file) in a temporary directory and creating a connection to it. The server contains a global browser variable which is an instantiated zombie browser object and has access to the stream via an argument.
• All of the Behat\Mink\Driver\DriverInterface methods are translated into JavaScript commands that are sent to the server. These interact with the global browser variable. Each command is stored in a buffer and then when stream.end method is invoked, are executed (using eval in the node JavaScript).
• The window object is available to your step-definitions and as such you can execute JavaScript and use jQuery etc to trigger events.
• The Zombie implementation has no concept of a viewport so resizing, scrolling etc aren't available, however depending on how the JavaScript under test is architected, you can use jQuery to trigger similar events if needed.
• The main drawback experienced with Zombie is internal limits. It seems to be limited to the size of the request it can parse and/or the time in which the server takes to respond. For slow performing pages/sites we've had to ensure the cache is enabled, CSS and JS aggregation are turned on and the cache is primed for the given path for Zombie to work. When it doesn't work, it fails silently with an empty dom.
• In terms of JavaScript testing I think there is an intermediate step we can take, we don't need Behat to use the Selenium/Zombie/Sahi et al drivers. To this effect I've added #2232861: Create BrowserTestBase for web-testing on top of Mink. This opens the door to Step 2: Add JavaScriptTestBase which uses a JavaScript capable Mink driver. My concern is that the Behat road is a long one and we could find ourselves twelve months down the road with no Behat, but also still no JavaScript testing. The Mink path is actionable now and also modernises the guts of WebTestBase in the process.
• In terms of cons of Behat, they've been touched on already by @jhedstrom - isolation, isolation, isolation. I second the 'clean up after yourself' notion but just don't see us replacing simpletest in entirety without some form of isolation and so I still think that leaves a place for TestBase's notion of a child site. Although we're making great progress in #2016629: Refactor bootstrap to better utilize the kernel which should hopefully open the door to HttpKernelTestBase.
• The other issue we've experienced with Behat in general is timeouts, but thats tuneable.
• In terms of adding our dependencies I agree - we need to pivot away from having /vendor in our repo. If you get the tar.gz, the packaging script can add those dependencies. If you're fetching with git, you can run composer. This also applies to the node modules required by Zombie etc. We add a packages.json/lock to our repo and thats it. Those who want to use the JavaScript testing framework need to worry about installing node and npm, these don't become a hard-requirement for Drupal. Our testing infrastructure then includes a composer install and a npm install. We make the node_modules configurable via settings, but default it to DRUPAL_ROOT/node_modules. If you're using global installs of node modules, (like testbot might), you can configure the path in your settings.
• In terms of running and collecting results, I see the docker work done by @jthorson, @nick_schuch, @ricardoamaro and @beejeebus et al in #2161175: Plan/create a Docker container for pifr deployment make management and maintenance easier has a next step of jenkins integration so I would advocate that we work towards a formal build process in core. This would entail a build.core.xml for consumption by Phing. Testbot installs Drupal in a subdirectory (drupal) so at the top level (outside /drupal) it could include a build.xml that imported build.core.xml definitions. These definitions would include build commands for testing simpletest tests, behat test, phpunit, automated visual regression testing - and anything else needed. All of these can produce output in junit format for consumption by Jenkins. With the Jenkins console, we could watch test runs as they execute - neat!

Drush isn't mandatory, just one of the available drivers for communicating with Drupal. The Drupal driver would be the way to go. Drush is used for the testing of the extension itself (see .travis.yml, and the bundled Behat features).

Yes, this means we'd add the Drupal driver to core, but the drush driver would stay in contrib.

I think we can handle isolation via a @tag on the scenarios, at least to start with. We know it's possible to do both isolated, and not-isolated tests, so we can figure that out later IMO.

I've personally used selenium+phantom but don't have a preference for that vs. zombie. The advantage of behat is apart from the test running environments and limitations in the browsers themselves, it's very little effort to switch.

Thanks for opening #2232861: Create BrowserTestBase for web-testing on top of Mink, following.

What's the need to refactor the Drupal driver out of DrupalExtension and plop it into core? I don't understand the benefits vs vendoring.

Core APIs change very, very frequently. We also have a requirement that 100% of core tests pass all the time (i.e. a failing test prevents commits).

That means that any change to core that would break the Drupal driver, would either need a simultaneous commit to the driver, or some sort of bc layer added. This applies to both the driver and especially DrupalExtension.

While 8.x has added a lot of external dependencies to core, we currently don't have any core dependencies where core counts as 'upstream' pulled in via composer.

I'd like to attempt to create a brief overview of the topics that needs to be discussed and the decisions that need to be made before Behat can become fully integration with Drupal, both on a core and on a contrib level.

1. Directory structure

   We need to define a standard for the placement of various required Behat files. This includes:

   • The BehatExtension. Questions: Should this be housed in core? If so where? Should it be downloaded separately with vendor files?
   • Subcontext files for each module. Questions: Where in the module directories should these be stored? What should the file naming convention be? Should we use PSR-0/4?
   • A yml config file. This may be required per-module and per-site (in a multisite install). Questions: should we define the Behat settings in existing module yml file?
   • .feature files for each module.
2. Implementation of subcontexts

   We need to define standards around how each module creates a subcontext, which should contain custom step definitions. Suggestion: Every subcontext should be an extension of the main DrupalContext.

3. Behat + Drupal bridge

   We need a tool that will aggregate all of the various Behat configuration settings and files provided by individual modules and provide them to Behat. A few requirements that come to mind:

   • Behat needs to know the locations of all .feature files provided by various, enabled modules
   • Behat needs to know the locations of all subcontexts provided by various, enabled modules
   • Behat needs to know the yml config settings provided by various, enabled modules
   • Behat should be aware of global and site level (e.g., in multisite setup) yml config settings
   • A drush command or a UI should be available for executing and perhaps viewing the status of these test
   • We should consider how to integrate this with various build tools, like Jenkins and Travis CI
4. Session Driver

   We need to determine which session driver will be used by default for Javascript Behat tests. Some available options are: Selenium2 and Zombie.

5. Determine details around implementing TestBot for these tests

I've started a sandbox fork of D8 that introduces Behat testing into core. It is not yet functional, but if you're interested in commenting or contributing, it's here: https://drupal.org/sandbox/madmatter23/2239057

Item 2 is no longer an issue: subcontexts are going away in Behat3, which is just a few days away from release at this point. The new suite-oriented Behat configuration may simplify our work in this regard.

Also, note that the WebApiContext I mentioned earlier is now a separate component, being completely redone for Behat 3.

I was recently patching the Drupal Behat Extension to work with D8's new session management functions, and got into a conversation with sun regarding the Behat Extension's Drupal driver, which bootstraps Drupal and manually opens a session via the Drupal API. He brought up a good point:

The moment you start a session in a test runner, you open the door to insanity (a.k.a. Simpletest's WebTestBase), because you're duplicating state between the test runner and the site under test... Simpletest has actually matured a lot in D8, but that duplication of environments still exists, and is one of the last things that causes plenty of problems... I'd highly recommend to not enter that path.

As it applies to the Behat Extension, this comment suggests that we should not bootstrap Drupal at all. That would mean not using the Drupal driver. All interactions with Drupal should occur via a RESTful interface. To quote sun again:

a BDD test should exclusively log in via the user login form only. Likewise, creating a user role, is equally POST /admin/config/people/roles/add, etc.pp. This kind of testing should have a _strict_ separation between the test runner and site under test + no shared state at all.

The primary argument against this approach, as I see it, stems from performance concerns. It's faster to create entities with a simple entity_create() call than it is to make a GET request for a form, fill it out, and POST data back. However, this could be somewhat mitigated by sharing fixtures (set up steps) between scenarios in a given Behat feature.

performance was the exact (misguided) idea that made us call API functions in the test runner (instead of performing requests) + inherently introduce a shared state between the test runner and the site under test due to that, which turned into a utter freakin' PITA over time.

This seems like an important topic to discuss before going to far with integrating Behat into core. Any thoughts?

I think it is OK to call into Drupal's API to do setup work like create entities. I dont think we have any choice. The REST API didn't get around to supporting all field types in core, and contrib will be worse. For example, we can't yet populate image/file fields using REST API.

I agree with Sun that all checking of functionality should happen via login form, link clicking, etc. I also abhor the shared state problem in Simpletest that Sun refers to. I just think we are falling into the same trap here. I guess I should look at the Drupal driver a bit closer.

Relying only on the UI is a problem because once outside standalone core development, in actual site projects, themes can build screens differently, and make the core tests fail. The Behat extension currently addresses this by allowing the redefinition of a small number of strings or selectors to customize expectations, but to use Behat on a larger would require this sort of customization to be much more extensive.

Also, BDD is not just for UI, but also applies to web APIs, where the "customer" expectations are different, and tests are not expected to rely on a HTML UI, but (typically) only on JSON or XML exchanges. The interesting thing is that these are independent from theme, and can probably be independent of language too (another problem with UI-based tests), and can provide a more solid, and possibly faster, basis to build test contexts (loading fixtures), than working through the UI steps. That's what we did in a recent project, BTW.

In a typical Given/When/Then sequence, the three predicates must be expressed at business-level, but there is no reason why they would have to be implemented as such, especially (IMHO) for the Given, which is a starting point from which actual checks are done, and should probably be done as efficiently as possible, while the other two are more representative of the actual behaviour to be tested. Of course, the fact that the steps implementing G/W/T are exactly the same may make things a bit difficult.

Awesome, thanks for starting the sandbox grashmash!

We want to work in such a way that will allow us to easily propose changes to Drupal 8 core when this project reaches a state suitable for incorporation. So, let's make sure that any changes to core itself are associated with an issue and that that issue is tagged with Behat. Once we do reach the point where it becomes necessary to patch core, let's start a distribution project so we can track the patches.

@jessebeach I've actually already modified core in the sandbox. A step has been added to the installation process that configures the necessary behat.yml file.

@fgm: "UI" was not strictly meant to mean "HTML UI" in #16. Focus was on avoiding shared state (which implicitly means that booting a parallel Drupal and calling API functions should be avoided) and thus keeping the test runner strictly separated from site under test.

In fact, in an ideal world, all fixtures would be set up by performing RESTful HTTP requests (via REST module). — However, I was told that the RESTful server capabilities in core do not support config entities yet (which probably present the largest chunk of fixtures). I don't know how easy or hard it would be to resolve that blocker.

grashmash, do you plan to do the bulk of the integration work in a module? Does that module exist yet?

@jessebeach I do not plan to house the initial integration in a module, although there may be a need for a module later on if we'd like to provide a UI. Currently, the plan is to for Behat code to live in two places: 1) in core .and 2) in the DrupalExtension library (which is not a module).

I've currently implemented the following additions to core:

• core/behat.yml (required config file)
• core/tests/features/Installation.feature (an example Behat test)
• core/tests/features/bootstrap/FeatureContext.php (required Behat class)

In order to properly follow PSR-0/4, we may need to move and rename FeatureContext.php.
install.core.inc has also been modified so that it sets up the proper configuration in behat.yml during the install process.

I've also modified composer.json so that it downloads a couple of required Behat libraries, including the DrupalExtension, which will live in core/vendor.

@moshe @sun

We may not actually need a REST API in core in order to avoid the shared state problem. Behat can make GET and POST requests to the UI via the Goutte driver (which leverages Guzzle). This will work, but has two distinct disadvantages: 1) This will significantly slow down testing given that each set-up step will need to request fully-rendered pages from Drupal. 2) This will a LOT more legwork to modify the existing DrupalExtension library, which is already functional using the bootstrap approach.

As I said before, I think it is fine to call into Drupal to do the GIVEN stop like setting up entities. It is just the WHEN/THEN steps that should be indepant of Drupal. I don't see how this violates the 'shared state' problem that Sun mentions. For starters, there is only one Drupal (at least in my head). The test runner is just the `behat` or `phpunit` unix programs. If we are talking about simpletest being the test runner, I'm no longer interested.

@moshe I'm inclined to agree with you, and I've already implemented the DrupalExtension so that it bootstraps Drupal and uses the existing GIVEN steps that utilize the Drupal API. Just wanted to make sure that all of the angles were considered.

re: #23, the Core modifications will be small and non-clashy, so in the interest of reducing friction, let's leave this as a Drupal sandbox for now and just make sure we merge in 8.x branch commits often. That will allow us to merge the change back into the 8.x branch later without conflict.

We can use the sandbox as the repository for an initial set of tests as well. Given a couple examples, I'll invite a wider audience to come in and propose patches to add coverage to features.

mtdowling pointed out on twitter that goutte which uses guzzle 3, we're using guzzle 4. I'm working on upgrading goutte to guzzle 4 as part of #2232861: Create BrowserTestBase for web-testing on top of Mink

What is the smallest step we can take now to allow us to start writing tests?

Could we work grasmash's sandbox into core in an inert way that developer's can leverage locally? This would allow us to start populating core modules with feature tests even though they won't be run on testbot. Then, as the Mink work matures, we can swap that in and hook up with testbot (/me waves wand).

The lionshare of the work will be writing feature tests and the sooner we can start with this work, the more coverage we'll have when we finally run this on the integration level.

With simpletest, we moved the contributed modules and it's tests (some of which had code comments in Polish etc.) directly into core wholesale.

Many of the tests were failing, or were broken by changes to core after they were added. Also there was no testbot support (at least patch-level testing) for some time between the core move and enabling it.

I'd personally be fine with doing similar here, but with one caveat:

Once it's feasible to enable test bot support for behat, we should do so. This means we need to be able to have 100% passing tests when that happens. A small set of passing tests that we can report back to issues when we break them, is a lot better than a large set of failing tests that we can't report on. Also trying to fix issues without automated marking of patches CNW is a losing battle given the rate of core commits, and it's worse the more tests there are. If we have lots of coverage that's OK, as long as we can easily disable failing tests.

On the Drupal driver. I personally would prefer to use REST and fixture sharing if we can. For one thing this would allow for using the core behat framework to run tests on non-local sites for client projects etc.

However given it works, I think using the Drupal driver for setup only is a good start and we can see how it goes down the line - as long as we don't switch between the drupal driver and mink back and forth it shouldn't be an issue with assertions.

After some discussion with catch, I think it's necessary to define the goals here:

1. Replace simpletest with Behat/(Mink|Goutte)
2. Use Behat + ?? for JavaScript Unit Tests
3. Use Behat + Selenium (or some other JS enabled extension) for JavaScript feature tests

Given that Simpletest works right now, replacing it is less of a concern compared to the zero coverage we have for JavaScript in core.

So, when I mentioned before, what is the smallest step we can take to start writing tests, I meant feature and unit tests for JavaScript. It's my understanding that grasmash's work in the sandbox will allow us to start writing feature tests for JavaScript.

My concern is that this work not conflict with larowlan's efforts to replace simpletest with a Behat/Mink combo.

The approaches are compatible.
Regardless, I'm blocked since we went to guzzle 4. You might be in a similar boat with the goutte behat driver.
See https://github.com/fabpot/Goutte/pull/140

Behat with Mink/Goutte is working fairly well with @larowlan's goutte port, my mink goutte driver port (linked above) and a bunch of additional patches and hacks. You can even configure your composer.json file to pick up our ports automatically like this:

{
    "require": {
        "drupal/drupal-extension": "*@dev",
        "fabpot/goutte": "dev-Guzzle4 as 1.0.1",
        "behat/mink-goutte-driver": "dev-Guzzle4 as 1.0.9",
        "symfony/dom-crawler": "2.4.*@dev"
    },
    "repositories": [
        {
            "type": "git",
            "url": "https://github.com/larowlan/Goutte.git"
        },
        {
            "type": "git",
            "url": "https://github.com/md-systems/MinkGoutteDriver"
        }
    ],
    "config": {
        "bin-dir": "bin/"
    }
}

(We have that in a module and use the composer_manager module to pick it up)

A few pieces are not yet available, the mink-extension package also needs some changes, haven't figured out what to do with that as it's an old version.

I've updated my sandbox, which is now functional. I've also added a screencast that provides a walkthrough of the changes and instructions for running the behat tests locally.

@Berdir @larowlan I have not encountered any issues with Guzzle 4 as of yet. However, I'm open to utilizing the Mink/Goutte port.

Mosche pointed out that this approach adds a new requirement that the core directory be writable at install, which nobody (myself included) wants. A couple of alternative:

1. Move core/behat.yml to sites/default.yml
2. Modify run-behat-tests.sh such that drupal_root and base_url are passed to behat at runtime (no need to write to behat.yml)

I've made a branch called no-install-step, which implements approach #2. However, I'm not certain which option is better. A quick comparison (please add your thoughts):

Keeping behat.yml in /core:

Pros:

• We don't need to add anything to install.core.inc

Cons:

• Users needing to override behat.yml don't have a good example of how to define drupal_root or base_url in a yml file

Moving behat.yml to sites/default

Pros:

• Users have a good example of how to customize behat.yml
• Our bash script does less and makes no assumptions. All values are explictly defined

Cons:

• We need to modify install.core.php

Note: The following is a copy/paste with liugh edit of comment #158 in #237566: Automated JavaScript unit testing framework as requested by @nod_.

Unit testing, integration testing and validation testing testing are three different beasts, looking for a single solution for all three of them is doomed to failure (or at least ugly hack and workarounds the limitations of the choosen tool in order to use it iun a way it wasn't designed for).

QUnit is a unit testing framework for JavaScript, like PHPUnit is for PHP. So if what is needed are integration and validation tests, something more than QUnit is indeed needed. IMHO, for unit testing in JS, neither a browser or an HTML page are required, but you may need to mock some DOM objects (ie. you don't run your tests over an actual site with real pages). For instance, a properly designed Drupal behavior should be testable by calling it over a set of fixture DOM elements (ie. context of the behavior) and asserting the wanted changes of these elements or their children.

Being a BDD framework, Behat is a more a validation testing framework. It tests an complete web application as a blackbox, not its individual components. So it could be used to validate that the JavaScript on a standard Drupal install works as expected. As a bonus, Behat is also usable to test behaviors defined by both PHP and JavaScript at the same time. An example of validation testing would be testing that a specific Views admin scenario create the expected view.

That leave us with integration testing, which is hard to implement over code that is not designed as loosely coupled components. Unit testing has the same kind of issue, but while it is still possible to unit test individual functions and classes by mocking their (hidden) dependencies, integration testing requires a known logic to assemble components together in order to be able to tests the various integration scenarios.

My understanding is that event in D8, Drupal's JavaScript code is not designed as components (even tightly coupled), making integration testing nearly impossible to implement. So focus on both unit testing (which the original issue #237566 was all about) and validation testing (which seems to be to point of this issue) will be more rewarding. Both will encourage more discipline and better quality in the JavaScript code, which should eventually make it possible to implement integration testing.

Given my comment in #36, the idea of using "Behat + ??" from @jessebeach in #31 for unit testing is, IMHO, a bad one. Even if using Behat for unit testing does not create issues, it will continue to blut the line between the three kind of testing and fail to encourage developer to design code with lously coupled component.

@mongolito404: Well said. Nice summary on concepts. We might want to consider to incorporate that almost literally into core API docs.

Because we went guzzle4 looks like we might need to use behat 3. Which will mean refactoring.
https://github.com/Behat/MinkGoutteDriver/pull/34#issuecomment-41444679See

FWIW, here is the temporary location for the Behat 3 docs, until the main site is updated to cover Behat 3 instead of 2.5: http://behat.readthedocs.org/en/latest/index.html

Note also http://facebook.github.io/jest/index.html

@larowlan: Jest is a unit testing framework (which look awesome, if only because it mocks the DOM). This issue is about using Behat for functionnal testing. Unit testing and functionnal testing (which I prefer to name validation testing) are different things, have different needs and are better served by different tools.

@mongolito404 yes I'm aware but #237566: Automated JavaScript unit testing framework was closed in favour of this, so this is our de-facto place for listing stuff.

As mentioned here https://drupal.org/node/2232861#comment-8778953

Codeception could be another alternative to Behat.

+1 for some review/discussion of codeception as an alternative to Behat.

Here is a first stab at integrating Behat with Drupal 8 core.

Some notes on the patch:

• A new Behat module exists at core/modules/behat
• A new script exists at core/scripts/run-behat-tests.sh
• A new dependency has been added under 'require-dev' in composer.json, but the packages are not included in this patch
• Feature tests have been written for Quickedit and Toolbar (thanks jbeach!)
• There are still a few todos left in this code, though the base functionality is present

Some notes on the module:

• This module will search for all .feature files located in /src/features for all enabled modules and register them as entities, to be executed via the run-behat-tests.sh script
• Custom FeatureContext.php files may be defined in /src/features/bootstrap for each implementing module
• Drush commands have already been written for this module, although they are not contained in this patch

To test this:

• Apply the patch
• Install Drupal
• composer update "drupal/drupal-extension"
• Enable the behat module
• Update public://behat.yml with your local machine's base_url
• Start selenium server
• cd core/scripts
• ./run-behat-tests.sh

Updating status.

Didn't review the patch, but want to raise early: The files should be located in ./tests/features/*.feature - i.e., within the ./tests directory of each module and outside of ./src directories as those are reserved for PHP code.

Resubmitting patch. I've removed views integration for now (since it was causing testbot failures) and I've moved .feature files to tests/features at sun's suggestion.

Unfortunately, a recent change to session handling in D8 core requires a few changes to the drupal-extension library before this can be merged. We'll need to wait for 1.0.3. This commit will fix it: https://github.com/Berdir/drupalextension/commit/bb7e705c31064d616421949...

Aaaand I'm attaching the patch.

Well, 16 failures errors is better than 61!

The following error occurs due to the fact that I haven't actually committed the new library, just added it to composer.json.
Class 'Behat\Gherkin\Keywords\ArrayKeywords' not found
This may be causing the module enable to fail, therefore triggering more test failures.

What's the proper procedure for updating composer dependencies in core patches?

@nod_ Created #2302545: Add Behat libraries to D8 core

Curious if we'd be able to use something like this to test interactions with more complex JavaScript, like interacting with CKEditor?

@Dave Reid: We use Behat to test TinyMCE in the Panopoly test suite. ;-) I think it should be possible to test CKEditor as well!

This would be huge.

I posted some concerns for discussion with Behat here: https://www.drupal.org/node/2302545#comment-8985123

JS testing would be huge, I agree with that.

@xjm: The title is misleading now. Behat is not restricted to JavaScript testing and is not a generic solution for JavaScrip testing (for instance, Behat is not the appropriate tools for JavaScript unit testing). Also, Behat can be used to test the result of PHP, and even more importantly, to test the integration of JavaScript and PHP code. See my comment in #36.

IMHO, if Behat is to be used (which would be great), it should not be discussed and designed as a JavaScript testing solution.

Retitle based on #36. We will test both PHP and javascript code to verify that it accomplishes its goals at a high level.

I hope to reroll this patch soon.

I had to update drupal/drupal-extension to get a working install of behat, a fix a few things in the patch. I'm attaching a reroll here.

I assumed this was about Behat 3? What is the reason to use Behat 2 instead of Behat 3?

Reading https://github.com/jhedstrom/drupalextension

Drupal Extension 1.0 supports Behat 2.4, and Drupal 6 and 7 (with Drupal 8 support being backported as it changes). Drupal Extension 2.0 aims to work with Behat 3, and focus on Drupal 8.

As Behat 3.0.0 is out since 2014-04-20 shouldn't we use it? Behat 2 latest isn't touched since 2014-04-26.

I came to this issue via #237566: Automated JavaScript unit testing framework. I'd like to know if there is any interest in using CasperJS instead of Behat for front end testing. In my experience, CasperJS is a much better tool than Behat (at least the 2.x version) for testing JavaScript interactions, and is also a lot faster than Behat. It's easy to install and learn. It doesn't have a Gherkin style syntax for writing scenarios but this is somewhat mitigated by adding comments at the top of each test file explaining what is happening.

Drush integration is also possible for CasperJS.

I'd be happy to provide proof-of-concept code, and work to get this implemented, if there is interest.

A benefit of using mink/behat is that you can do more than just frontend testing, you can control the back end using the api.

Another benefit us that it can control a lot of browsers.

@attiks CasperJS can interact with the backend via Drush, which I think would be sufficient for this scope. It can also drive WebKit (via PhantomJS) or Gecko (using SlimerJS), which I also believe would be sufficient for what we are discussing here.

##6 I know CasperJs, but it is limited to those 2 browsers, and I hope that once we have this system for core, we will use it to test on as much browsers as possible.

There are/were to many javascript bugs in Drupal 7, most only showing on some combo of browser and OS. Since Drupal 8 contains even more javascript we (developers as well as users/end users) can only benefit of advanced front end and integration testing.

You're right about the unit testing part, we need to handle that as well, but there are other tools that we can use.

PS: We use a combination of jenkins, behat and phantomjs to do automated frontend testing.

... I hope that once we have this system for core, we will use it to test on as much browsers as possible. ... There are/were to many javascript bugs in Drupal 7, most only showing on some combo of browser and OS.

@attiks that's a very good point and this alone probably outweighs the benefits of using CasperJS.

@kostajh Please see #36, #61 and #63, Behat usage is not for considered for JavaScript unit testing (as reflected in the issue title).

Well.. I don't know what the status of this issue but i have something that may help this issue - https://github.com/roysegall/behat.

What happened to #44 and #45 mentioning codeception? See http://codeception.com/docs/01-Introduction . Although it's still PHPunit :( but it's PIE. Also, should I repost here my concerns over the whole testing world becoming phpunit based from #2469731-5: Document when to use BrowserTestBase or is linking enough or what's the best place to talk about this?

#2469731: Document when to use BrowserTestBase is probably sufficient for now, we don't need to discuss this in two issues.

@pfrenssen do you see any replies to those concerns there? I don't.

Here is rerolled patch with DrupalExtension 3.x (because obviously 1.x is outdated and pretty limited to huge projects when separate independent contexts are required)

All errors are missed library related, should I add libraries to the repo to check tests?

1. +++ b/core/modules/behat/config/behat.yml.dist
   @@ -0,0 +1,30 @@
   \ No newline at end of file
   

   No newline at end of file

2. +++ b/core/modules/quickedit/tests/features/Quickedit.feature
   @@ -0,0 +1,13 @@
   +  ¶
   

   Whitespace

Just a quick question to the patch developers:

1. +++ b/core/modules/behat/behat.install
   @@ -0,0 +1,26 @@
   +  $behat_conf = Yaml::parse(DRUPAL_ROOT . '/' . \Drupal::moduleHandler()->getModule('behat')->getPath() . '/config/behat.yml.dist');
   

2. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   +    $system_path = DRUPAL_ROOT . '/' . drupal_get_path('module', 'behat') . '/tests/features/bootstrap';
   

Which function do you prefer to get the path to the module? In the patch are two functions used: \Drupal::moduleHandler()->getModule('behat')->getPath() and the good old drupal_get_path('module', 'behat')

Behat has it's own command-line utility (similar to PHPUnit), so I'm not sure why we'd need to add and maintain a module that provides a UI. I think it would make more sense to keep the behat.yml.dist file in a similar location to phpunit.xml, and the tests under the core/tests/features directory (and in modules, MODULE/tests/features).

Agreed. We've had numerous problems with the Simpletest UI already, especially in combination with PHPUnit. If we discover we need additions to the third party (command-line) tools we use, we should try to fix those upstream instead.

BTW #2232861: Create BrowserTestBase for web-testing on top of Mink is fixed now!

For now the this a reroll, but I think we should refocus on the longrun. More tomorrow ...

1. +++ b/core/modules/behat/behat.install
   @@ -0,0 +1,25 @@
   +  drupal_set_message('behat.yml has been written to the public files directory. Please update it with the correct base_url.');
   

   The $base_url is set above in the $behat_conf.

   So what's the message about?

2. +++ b/core/modules/behat/behat.module
   @@ -0,0 +1,19 @@
   +      'description' => t('Grants all permission for behat.'),
   

   I don't understand this

   'Grant all permission for behat'

   It does not reflect the purpose for administer Behat.

   Behat should be uppercase.

3. +++ b/core/modules/behat/config/behat.yml.dist
   @@ -0,0 +1,30 @@
   \ No newline at end of file
   

   No new line

4. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   + * Discovers and registers all modules implementing hook_behat_info().
   

   Where is behat.api.php describing hook_behat_info()?

5. +++ b/core/modules/behat/tests/features/bootstrap/FeatureContext.php
   @@ -0,0 +1,196 @@
   +   * @When /^(?:|I )click the element with CSS selector "([^"]*)"$/
   

   Are we using Behat 2 syntax? /^ and $/ are iirc Behat 2 syntax.

6. +++ b/core/modules/quickedit/tests/features/Quickedit.feature
   @@ -0,0 +1,13 @@
   +  ¶
   

   Whitespace

I concur with @jhedstrom and @Xano in #83 and #84. I don't see the need for a UI, except to maybe configure behat.yml and the environment since that can vary. But even that might best be packaged as a Drush command or something.

I totally agree, just wanted to reroll so people can attention more.
Regarding behat, I think it should just work like any other behat usage, so using the behat binary ideally.

I took a quick look through the patch and found some relatively minor issues and nitpicks. I'm sure there are larger issues to deal with but my head is not far enough into this to raise any major objections :)

1. +++ b/core/modules/behat/behat.install
   @@ -0,0 +1,25 @@
   +  $behat_conf = Yaml::parse(DRUPAL_ROOT . '/' . \Drupal::moduleHandler()->getModule('behat')->getPath() . '/config/behat.yml.dist');
   

   I thought DRUPAL_ROOT was deprecated/replaced by the app.root container parameter?

2. +++ b/core/modules/behat/behat.module
   @@ -0,0 +1,19 @@
   +function behat_permission() {
   +  $permissions = [
   +    'administer behat' => [
   +      'title' => t('Administer Behat'),
   +      'description' => t('Grants all permission for behat.'),
   +      'restrict access' => TRUE,
   +    ],
   +  ];
   +  return $permissions;
   +}
   

   This looks like it can go in a static .permissions.yml file.

3. +++ b/core/modules/behat/config/behat.yml.dist
   @@ -0,0 +1,30 @@
   \ No newline at end of file
   

   D'oh. :)

4. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   + * Contains behat api implementation.
   

   s/behat api/Behat API

5. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   +  $module_handler = \Drupal::moduleHandler();
   +  $modules = $module_handler->getModuleList();
   

   Nit: Can be one line.

6. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   +  // If this module has been registered, remove its scenario registrations.
   +  if ($registered) {
   +    behat_deregister_module_scenarios($module);
   +  }
   

   Wat? This seems odd to me -- a function to register a module and its scenarios will deregister scenarios? Maybe this should be removed, or at least commented better.

7. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   +  $system_path = DRUPAL_ROOT . '/' . $module->getPath() . '/tests/features';
   +
   +  return $system_path;
   

   Can be a single line, and ditto with DRUPAL_ROOT no longer being a thing.

8. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   +    $entity = entity_create('behat_scenario', $data);
   

   Scenario::create($data)->save()

9. +++ b/core/modules/behat/includes/behat.inc
   @@ -0,0 +1,528 @@
   +  // @todo return somesthing here, particularly on failure.
   

   Number of scenarios registered, perhaps? IMO it should throw an exception on failure.

10. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +  $query = 'SELECT bsid FROM {behat_scenario}';
    +  $query .= ' WHERE module=:module LIMIT 1';
    +  $result = db_query($query, array(':module' => $module->getName()));
    +  $record = $result->fetchObject();
    +
    +  return (boolean) $record;
    

    This oughta be using DBTNG's fluent syntax (or better yet, an entity query), not db_query().

11. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +  $query = 'SELECT bsid, location FROM {behat_scenario}';
    +  $query .= ' WHERE module=:module';
    +  $result = db_query($query, array(':module' => $module->getName()));
    +  $registrations = $result->fetchAllKeyed(0, 1);
    

    Ditto.

12. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +  $entity_ids = array_keys($scenario_registrations);
    +
    +  entity_delete_multiple('behat_scenario', $entity_ids);
    

    \Drupal::entityManager()->getStorage('behat_scenario')->delete($scenario_registrations)

13. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +  $behat_params = parse_url(getenv('BEHAT_PARAMS'));
    +
    +  return $behat_params;
    

    Nit: Can be one line.

14. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +  return putenv("BEHAT_PARAMS=$behat_params_value");
    

    I don't like that this function returns a boolean to indicate failure. IMO, failure to set the environment correctly is a severe enough problem that an exception should be thrown.

15. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +    /*
    +    if (variable_get('behat_log', 'filesystem_database')) {
    +      list($scenario_file_path, $scenario_line) = explode(':', $scenario->location->value);
    +      $result_file_location = $out_dests['junit'] . '/TEST-' . basename($scenario_file_path, '.feature') . '.xml';
    +
    +      // Log results to database.
    +      $junit_result = behat_parse_junit($result_file_location);
    +      // @todo Complete logging function.
    +      //behat_log_scenario_result($junit_result);
    +    }
    +    */
    

    Why is this commented out? A @todo comment may be a good idea :)

16. +++ b/core/modules/behat/includes/behat.inc
    @@ -0,0 +1,528 @@
    +  $scenarios = entity_load_multiple('behat_scenario');
    

    Scenario::loadMultiple()

17. +++ b/core/modules/behat/src/Entity/Scenario.php
    @@ -0,0 +1,89 @@
    + *   admin_permission = "administer behat scenario",
    

    s/scenario/scenarios

18. +++ b/core/modules/behat/src/Entity/Scenario.php
    @@ -0,0 +1,89 @@
    +  public static function baseFieldDefinitions(EntityTypeInterface $entity_type) {
    +
    

    Nit: There's an extra line here.

19. +++ b/core/modules/behat/src/Entity/Scenario.php
    @@ -0,0 +1,89 @@
    +      ->setDescription(t('The Behat log id the most recent run for this scenario. NULL if database logging is disabled.'))
    

    Nit: s/id/ID

20. +++ b/core/modules/behat/src/Form/SettingsForm.php
    @@ -0,0 +1,73 @@
    + * Configures behat settings for this site.
    

    s/behat/Behat

21. +++ b/core/modules/behat/src/Form/SettingsForm.php
    @@ -0,0 +1,73 @@
    +  public function __construct(ConfigFactory $config_factory) {
    +    parent::__construct($config_factory);
    +  }
    

    Why do we need this?

22. +++ b/core/modules/behat/src/ScenarioStorage.php
    @@ -0,0 +1,27 @@
    + * Defines a Controller class for scenario entities.
    

    Storage handler, not controller class.

23. +++ b/core/modules/behat/src/ScenarioStorage.php
    @@ -0,0 +1,27 @@
    +   * We don't need to store behat scenario entities as they will be
    +   * automatically recreated during next install.
    

    I thought it was unkosher to add extra stuff after an @inheritdoc?

24. +++ b/core/modules/behat/tests/features/bootstrap/FeatureContext.php
    @@ -0,0 +1,196 @@
    +   * @When /^(?:|I )click the element with CSS selector "([^"]*)"$/
    

    I could be wrong, but IIRC aren't most if not all of these step definitions rendered unnecessary by MinkContext and DrupalContext?

25. +++ b/core/modules/behat/tests/features/bootstrap/FeatureContext.php
    @@ -0,0 +1,196 @@
    +  public function getNodeFromUrl() {
    +
    

    Nit: Extra line.

26. +++ b/core/modules/quickedit/tests/features/Quickedit.feature
    @@ -0,0 +1,13 @@
    +Feature: Quick edit Link
    +  Quick edit link displays quick edit link to edit content.
    +  ¶
    

    The blank line has extra white space.

27. +++ b/core/scripts/run-behat-tests.sh
    

    Presumably (hopefully) this could eventually be merged into run-tests.sh or whatever.

#88.1: Fixed. Added also translation for that string and alternative message if the behat.yml writing fails.

#88.2: Fixed and moved to behat.permissions.yml

#88.3: Fixed.

#88.4: Seems that the discovery is not based on that hook. Removed the reference.

#88.5: Don't know. Not addressed.

#88.6: Fixed.

+++ b/core/modules/behat/src/Form/SettingsForm.php
@@ -0,0 +1,73 @@
+  /**
+   * @param \Drupal\Core\Config\ConfigFactory $config_factory
+   *   The factory for configuration objects.
+   */
+  public function __construct(ConfigFactory $config_factory) {
+    parent::__construct($config_factory);
+  }

Removed.

+++ b/core/modules/behat/src/Entity/Scenario.php
@@ -0,0 +1,89 @@
+ *     "list_builder" = "Drupal\Core\Entity\EntityListBuilder",
+ *     "view_builder" = "Drupal\Core\Entity\EntityViewBuilder",
...
+ *   links = {
+ *     "canonical" = "behat.scenario_view",
+ *     "edit-form" = "behat.scenario_edit",
+ *     "admin-form" = "behat.scenario_settings",
+ *   }

Are we gonna expose and handle scenarios on UI?

EDIT: And I think behat.inc should be moved to a manager class.

I think we first should think about what we actually want ... rolling a patch was a bad idea probably.

Do we really want the UI? I think absolutely no. IMHO you don't get any advantage out of having one, or do you?

I agree. In #92 I removed the links from the entity. There's only the settings UI left.

Do we really want the UI? I think absolutely no. IMHO you don't get any advantage out of having one, or do you?

I'm a bit late jumping into this issue, but I'm opposed to having a UI for Behat. It's not a direction I think we should be going, given the way core in which core is already using PHPUnit and moving away from SimpleTest. Behat is already very usable from the command line, and bolting on a UI would add complexity that provides no advantage to core developers. With Behat itself in core (along with the Mink and Drupal extensions), a UI is the kind of thing that would be well-suited to contrib. But it's not something core developers should spend time on.

Just my $0.02.

Fixed almost all from #91

The issue summary references an offline conversion saying, lets use Behat but i'm still yet to see an issue that outlines why that's a good idea. Are we doing Story BDD now? The IS mentions that "lots" of core developers know/use Behat, that may be true but it's still a subset of core developers who know PHP, so we're making testing less accessible to our newcomers.

It gives us access to real browsers and phantomjs via mink

It seems people want to test JS, and that's great but we can test our JS enabled code by writing PHP tests via BrowserTestBase and using whichever browser plugin we want. We don't need and we shouldn't use Gherkin unless we have a need.

This wouldn't be a replacement for SimpleTest, but likely all the SimpleTests using the browser could be moved over.

This just makes me cringe. Behat isn't designed to replace all of what we test with ST. BrowserTestBase is as far as we needed to go. I think we need Mink, not Behat. http://elnur.pro/behat-and-mink-are-not-meant-to-be-together/

@benjy in #100:

After reading your comment I followed your link and that started a journey through a lot of blog posts that talk about the intent of Behat, it's proper place within a project, and alternative strategies of testing that do not use Behat.

From those discussions I've gathered a few key points:

1. Behat is a communication tool. It helps development by including the people who have knowledge of the business rules of a project to help set the language and grammar used in tests. By keeping these consistent through to testing everyone can communicate the same way about the project.
2. Behat isn't the only thing that can do the kinds of testing we would use Behat for. PhpUnit could be use Mink to test the front end of Drupal. PhpUnit could do integration tests on the output of Drupal's web service. PhpUnit could of course test the "units" as well. Having Drupal test all of these things would have the benefit of having one test platform to test all of these things.
3. Using Behat at a granular level (being too verbose / non-DRY with the wording of the tests) may defeat the purpose of using Behat. Our goal when using Behat is to create user stories that drive tests. Directly converting our functional tests to Behat would very likely product the kind of too verbose tests that http://elnur.pro/use-the-domain-language-in-bdd-features/ warns against.

All that said:

I still see the value of using Behat for testing Drupal. Because:

• We would be forced to document all the business rules of Drupal in an easily accessible way.
• We would provide new developers an easier-to-get-into way of helping with tests, due to the fact that new functionality will need a human-readable user story and the code to help drive the test.
• In the end the effort would provide a consistent way of thinking and talking about Drupal's functionality that would improve how new features are consistent with that.

So imagine this:

Road #1: We include Mink but not Behat. We write all the tests we need in order to cover all of Drupal's frontend and javascript using a combination of PHPUnit and Mink (and drivers).

Benefits:
For all the people who are writing PHPUnit tests now, a consistent platform, confidence, and speed of development as they create tests to cover JavaScript behaviors.
We eventually convert all the Simpletest tests to PHPUnit.
Perhaps we eventually create objects to make it "easy" to write these kind of frontend tests. (that sounds like us)
In the end, we'll have one test framework to cover all the kinds of tests in Drupal.

Detriments:
You have to know PHPUnit to participate.
You have to be comfortable with code and the level of complexity that exists in our tests to make progress.

Road #2: We continue to expand PHPUnit's coverage to Unit testing, functional testing, and integration testing. We also use Behat to cover all the user stories of Drupal.

Benefits:
We document all of Drupal's user stories
We have a great reference for how to turn user stories into tests for other Drupal projects
We continue to benefit from PHPUnit's tests, it continues to be the principal source of truth on whether something has complete test coverage or not.

Detriments:
We now have two tests frameworks to maintain
We'll need people to write the Behat tests, who may not be the same people who write the PHPUnit tests
Therefore the tests may get into conflict (but we can make sure they don't. We're not robots)

So that's my whole thought about this. I see the value of going through the exercise of coverage the business rules with Behat tests. So we can better communicate about minutia of it's functionality. I also see the benefit of converting our functional tests into PHPUnit tests (not Behat). I'm interested in everyone else's thoughts.

Thanks @cosmicdreams.
Fwiw we're already well down road 1, BrowserTestBase has been in for some time and JavaScriptTestBase is close.

@cosmicdreams, glad you found some of those posts interesting, I think as larowlan already mentions, JavaScriptTestBase is the way forward.

Maybe this might help people to be confused less about what currently we try to achieve:

The current testing plan

Drupal iterated on the great success of simpletest in Drupal 7 to improve the following points:

a) Enlarge the test coverage
b) Improve the test developer experience

Therefore we now have a 3 level testing system:

1. Unit tests
Unit tests are there only for testing a particular component, like a class. Those
are tests using PHPUnit's \PHPUnit_Framework_TestCase

2. API tests
In order to test the complexity of Drupal its not enough to test each individual
component but you also need to test the ingration between various subsystems on sort
of a working Drupal. The kernel test set ups a bare running minimal Drupal, which
allows you to call APIs, but not to visit the actual site.

In order to write those you need to extend KernelTestBase.
There is an old one and a one new. I would recommend the new one (\Drupal\KernelTests\KernelTestBase

3. Integration/Functional tests
One level higher you want to browse the actual site and check whether the behaviour is
as expected. This field is currently still in the flux.

There is \Drupal\simpletest\WebTestBase which is a safe choise based upon simpletest. We try to replace that slowly over time by \Drupal\simpletest\BrowserTestBase which uses a browser abstraction library called Mink.

This would allow us to not only use a PHP based "browser" like WebTestBase does, but one day use an actual browser or PhantomJS as browser, so we can test javascript behaviour as well. This is researched on #2469713: Step 2: Create a JavaScriptTestBase using PhantomJs Driver/Binary.

4. BDD
On an even higher level we should document what Drupal does. Using a Business readable DSL allows us to document it in a implementation agnostic way.

Doing that using Behat would allow us to execute that behaviour documentation as well, though its not a replacement for any of the previous levels, rather a supplement.

In case we have clear levels on the code architecture site of things, it also becomes easier to understand on which level you want to target your tests on.

This seems like this should be documented somewhere.

If tests have this natural categorization could we make it easy for developers to know what kind of test they're looking at / organize tests into folders / annotate tests so that their purpose / job is well known?

4. BDD
On an even higher level we should document what Drupal does. Using a Business readable DSL allows us to document it in a implementation agnostic way.

I think BDD/Behat often become interchangeable but to clarify BDD isn't 1 level higher than functional tests. BDD is just a way of testing, often using Ubiquitous Language to drive the tests. You can just as easily use SpecBDD to unit test object behaviour.

However, I think what you were suggesting is that we use Behat and in-turn StoryBDD to provide a high level description on the features in Drupal, +1 to that.

I think BDD/Behat often become interchangeable but to clarify BDD isn't 1 level higher than functional tests. BDD is just a way of testing, often using Ubiquitous Language to drive the tests. You can just as easily use SpecBDD to unit test object behaviour.

Yeah I see your point, this is right. SpecBDD indeed can be used as an alternative of unit testing. I think though at least for now we should try to limit the options in core, given there are already quite a lot of them

Guess we can move this safely to 8.2.x

In trying to answer @cosmicdreams visiting https://www.drupal.org/documentation gives no clue.
- We do have https://www.drupal.org/phpunit (level 1)
- Searching on https://www.drupal.org/search/site/webtest?f[0]=ss_meta_type%3Adocumenta... gives API docs :-(
- https://www.drupal.org/project/behat_testing

@dawehner where can we put your #105 ?

This is interesting issue. Possible Drupal 8.2.0?

I think BDD/Behat often become interchangeable but to clarify BDD isn't 1 level higher than functional tests. BDD is just a way of testing, often using Ubiquitous Language to drive the tests.

BDD is actually not just a way of testing, BDD is a software development process. Using tools designed for BDD to just execute some tests is not BDD. Off course, you can use BDD tools anyway you want. There is nothing stopping you from using an icecream truck as moving truck when changing apartment, but it's not really designed for that and you shouldn't call that selling icecreams.

BDD is actually not just a way of testing, BDD is a software development process. Using tools designed for BDD to just execute some tests is not BDD. Off course, you can use BDD tools anyway you want. There is nothing stopping you from using an icecream truck as moving truck when changing apartment, but it's not really designed for that and you shouldn't call that selling icecreams.

Can't agree more with that statement.

Here is a document a couple of people have been working on to clarify what behat is about and what it is not: https://docs.google.com/document/d/17raktxD51MP0WUvqtNQFsCzevYgDNkvl7jOu...

Yes I think we have plenty with phpunit + mink and should leave behat/gherkin/BDD for now.

That could be revisited, but we really need to bring our existing tests up-to-date and rationalise them first. So postponing this for now on #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) (but really on further follow-up once the initial conversions are done).

It could be interesting to use gherkin as a pure language to describe features we want to add to core, but skip the testing aspect of it.
This helps already to do less bikeshedding once we are on the implementation level.

Yes, I think it would be great to start using Gherkin as a standard format for describing features and user interactions, which we could leverage in the future for testing if appropriate.

That makes sense if you look at Gherkin / Behat as a communication tool

Moving to core ideas as a plan.

What's the latest thinking around this? Is there another active thread for this somewhere maybe?

I think everyone involved in the phpunit initiative doesn't believe in behat as a testing tool for Drupal.

Behat is a tool to talk to enable a common business language with your client for your thing you are building. You can then validate this business language by converting these constrains into running code.

What Behat does't solve is to provide full test coverage for the full complexity of the world. Writing the step definitions is not easy to maintain and you end up writing basically a lot of code, so you could just write code only tests in the first place. For example having context is a really tricky topic in behat.

On top of that I think having edge case testing is really hard in Behat, given that you again need to define a lot of custom step definitions to be handle to deal with them, which ends up in a maintenance nightmare. This is much less of a problem for client projects / products, because you have way more limits involved. You have concrete and endless usecases, while in core we just enable people to build things with our work.

Don't get me wrong, I personally believe that the business language aspect of Behat/Gherkin/BDD is great and super useful! I could imagine having something along the lines of userstories expressed in issues to describe bugs, features wishes etc. would be beneficial, but I don't believe that converting those into runnable code would fit for the complexity of Drupal.

+1.

Behat is great for actual projects, to test the integration of all the modules you use, their configuration, the specific case you chose to use them in your project and a limited amount of test scenarios.

But it's not well suited to do the kind of tests we have to do in Core, with many variations, very specific assertions and lots of API calls and so on.

At best, it could be used to test the new example install profile that we want to put in core, as that basically is a specific project. But I think it would be a lot of dependencies for just that and doing those tests isn't that much harder to do directly against Mink. A few years ago, when this was started, we didn't have that. Now we have browser and Javascript tests through Mink.

I think we can close this.

Well put dawehner. +1 to close this issue.

I like @Berdir's suggestion that the example install profile include Behat tests.

Such tests may not be necessary to ensure the install profile functions correctly, but one of the great things about Drupal is that you can get started without knowing how to code, and learn to code later. Behat is a way to do the same thing with testing for the site you are building, but Behat documentation isn't very good for people who are just trying to write a couple scenarios for the first time, so having good, working examples for a standard Drupal install profile (a way to check "Did I break anything?") is a great way to help people who are getting started with Drupal help themselves debug problems.

Now that we have a fantastic installation profile in core, shall we revisit, as suggested by Berdir, the advantages of getting Behavior-Driven Development tests in core mentioned in this thread?

* document Drupal's user stories— high level descriptions of the features in Drupal (and/or Umami) (cosmicdreams / Benjy)
* have a great reference for how to turn user stories into tests for other Drupal projects (cosmicdreams / ptmkenny)

The Drupal 8 testing documentation mentions Behat as an addition to the three kinds of tests in D8 core. We're clear on not trying to replace any of those tests, but adding human-readable subject matter tests that can also be run.

The first step would be to write these high-level business-logic acceptance tests— a virtue in their own write (per dawehner).

Perhaps a good place to start would be the Umami user stories (not certain that's the final or most complete list).

Later we can handle wiring up the human-readable gherkin to tests which can be automated, and work out if or how to break up Behat features (collections of scenarios) among installation profiles and modules or feature modules.

So as a first step, to clarify, you're thinking we should ensure that the spreadsheet you listed is really the complete list of user stories for hte Umami installation profile?

I think it would also be pointed out that what we're not saying should be done at this time is making the running of behat tests a requirement of the testbot. That we're just asking for the inclusion of the behat tests so that it can be provided to developers as a complete example of how one would write behat tests for an installation profile. Like, a demonstration of the level of effort it takes to both provide and maintain the tests.

Is that right?

Or,

If not specifically Behat, a suite of functional tests that cover every user story.