Maintaining a set of BDD tests

Test sets

Automated BDD tests on Drupal.org are grouped in the following folders:
casestudies
changerecords
community
docs
forums
git
issues
maintainers
marketplace
other
planet
profile
project
releases
search
security
user

Test runs

Automated tests run daily(?) against our testing services
http://git7site.devdrupal.org
Rebuilt daily at 8:00 UTC. Takes a couple of hours, during which test server is not accessible.

Daily test output: http://bddtest.drupal.org/test-output/maint

If you want to run the same set of tests on your local machine, you’ll find the tags in the doobie/maintenance folder. For example, to run the tests you see at http://bddtest.drupal.org/test-output/maint/user, look in maintenance/user. You’ll see user&&~wip&&~profile&&~git&&~project

Then the tests can be run with the command:
bin/behat --tags=”user&&~wip&&~profile&&~git&&~project”
Notice the quotation marks. Tests will run without those, but the logic will be different, so be sure to include them.

How to maintain a set of tests

1. Look at failures daily-ish
2. Try to determine if the problem is with the site or the test.
3. Open an issue in the Doobie queue for the failure. See https://drupal.org/project/issues/search?text=&projects=&assigned=&submi... to look for issues that may already be open.
   • Issues should be tagged Drupal.org D7, qa
   • Include the path to the test and the page where the problem can be observed
4. The issue should indicate:
   • Failing test (when you’re not sure)
   • Broken site (if that seems to be the problem)
   • Test needs update (if the values in the .feature file are wrong)
   • Step definitions (if the Step definition needs update)
5. Link the issues into their “Featureset review issue” by copying the issue number, editing the Feature set review issue, and including it with the dynamic notation: #12345: How do I log in as admin?