The purpose of BDD

Please ensure these all have the Given/When/Then structure before marking for review. I shared a link to a checklist we made a while back that should help with the myriad things we need to check for feature files before they're really done.

I think there's a fundamental misunderstanding of the purpose of behavior-driven development tests that is going to disrupt the value of these tests to developers.

Tests are being introduced which are checking that content on a page is "what it should be." Here's one example: http://drupalcode.org/project/doobie.git/blob/e4b9f49:/features/drupalor...

The only application behavior being tested here, really, is that when a privileged user clicks edit and puts in text and links, then text and links are displayed (e.g. not filtered out by the input filter). That's what the app is doing.

What the test is checking, though, is whether a human entered/changed the data the test wanted them to have entered. It is absolutely normal for this text to change, there is no process to track when a change will happen or has happened or whether anyone decided it was actually a good idea.

Including this sort of test in this project adds time to the complete test run, delaying the feedback that developers need before a code deployment and when failures occur, distract from the issues at hand.

I feel it's imperative that such tests are easily separated from the actual BDD tests in some way. Ideas include, but are not limited to:

a) an independent branch
b) an independent project
c) tagged in a way that permits exclusion

I think if we clarify when and how they'll be run, what they'll influence, who will review and update failures on what time schedule, etc. we can figure out how to best decide where they live.

I completely agree with eliza411. The focus with BDD is to test the *behavior* of the site. Submitting a node and then having that node properly display is the behavior being tested here, but as a behavior that can and should be tested generically not in a way specific to the content. To eliza411's point this is going to cause pain for developers and test maintainers because it means that failing tests don't necessarily mean behavior changes or code bugs.

It seems clear that verifying some of the content is extremely high priority and that we want to verify that it is displaying how and where we want it to, but I still agree that this should be somehow distinguished from our functional tests. I think our behat/mink toolchain is probably a good set of tools to perform these content tests but we should be clear about the fact that they are content tests and not behavior tests. Any of eliza411's options could work:

A. an independent branch

I don't like this option as much because it means we have separate branches to keep in sync indefinitely and they're actually responsible for slightly different sets of functionality. I think this will add a lot of conceptual overhead and maintenance cost over time.

B. an independant project

I think I like this option the most because it helps us draw a clear distinction between the stuff that is content specific and the stuff that is functional. Ideally I should be able to test the functional stuff with a fresh install of a drupal.org install profile (we may never get there, but ideally our tests create everything they need to verify for our real *behavior* tests). Then we have a separate thing that makes sure that with the full Drupal database our needs are met and specific content is what and where we want and that any unscheduled change is captured.

C. tagged in a way that permits exclusion

This is a fair compromise. It doesn't have the clear separation that an independent project has, but it does let us share step definitions specific to drupal.org between the two projects. With this approach we can still exclude the tests from execution when they are content specific (and vice versa). It would be great if we could even set a default to have behat ignore content specific tests unless told otherwise.

I think we should have a scheduled *separate* job that runs content specific tests so that we can clearly see when there are unexpected content failures and when there are unexpected behavior failures. These are the concerns of separate but overlapping teams.

I think I'd still prefer B for clean separation, but C may keep the maintenance and complexity to a minimum.

I'd say if there are folks that are worried those pages could change or break in an unplanned, unapproved or unexpected way then let's test for them (in a way somehow isolated from our functional tests) but if we aren't particularly worried we shouldn't add unnecessary tests out of any sense of obligation to test ALL THE THINGS. :-)

Scenarios were ultimately tagged with @content

.