Running tests through command-line with run-tests.sh

Command-line test execution - Linux/Mac

This section is for the those that have the desire and/or the need to run the test-suite by command-line. This is often needed when one wants to integrate 3rd-party continuous-build tools, for example. I am hoping this page will collect info and tips on the manual aspects of testing, when one is doing its small-scale and only wants to test one or a few modules.

For those looking for info on the GUI way of running Simpletest, head for the Simpletest tutorial; lower down in that tutorial and you will find some information for running tests through the Drupal user interface. The two important links for command-line commands accessing Simpletest through Drupal interface are admin/config/development/testing/ and admin/config/development/testing/results/

Running the tests through the command line in Drupal is quite easy with the help of run-tests.sh, located in /scripts. One can easily get the test-suite running by going to the root of the site directory, as web-server user and typing php scripts/run-tests.sh.

If you don't feel like typing /scripts/run-tests.sh all the time, symlink the script to your ~/bin directory, assuming you have an existing bin directory that is in your path:

#Drupal 8
ln -s core/scripts/run-tests.sh ~/bin
# Drupal < 8
ln -s scripts/run-tests.sh ~/bin

Running the script with no arguments will bring-up the help page, with a list of the available options.

Here is the run-down, shamelessly ripped from run-tests.sh, note Drupal 8 users need to add core/ in front of scripts:

www-data@dev:/var/www-drupal-8-head$ php core/scripts/run-tests.sh

Run Drupal tests from the shell.

Usage:        run-tests.sh [OPTIONS] <tests>
Example:      run-tests.sh Profile

All arguments are long options.

  --help      Print this page.

  --list      Display all available test groups.

  --list-files
              Display all discoverable test file paths.

  --list-files-json
              Display all discoverable test files as JSON. The array key will be
              the test class name, and the value will be the file path of the
              test.

  --clean     Cleans up database tables or directories from previous, failed,
              tests and then exits (no tests are run).

  --url       The base URL of the root directory of this Drupal checkout; e.g.:
                http://drupal.test/
              Required unless the Drupal root directory maps exactly to:
                http://localhost:80/
              Use a https:// URL to force all tests to be run under SSL.

  --sqlite    A pathname to use for the SQLite database of the test runner.
              Required unless this script is executed with a working Drupal
              installation that has Simpletest module installed.
              A relative pathname is interpreted relative to the Drupal root
              directory.
              Note that ':memory:' cannot be used, because this script spawns
              sub-processes. However, you may use e.g. '/tmpfs/test.sqlite'

  --keep-results-table

              Boolean flag to indicate to not cleanup the simpletest result
              table. For testbots or repeated execution of a single test it can
              be helpful to not cleanup the simpletest result table.

  --dburl     A URI denoting the database driver, credentials, server hostname,
              and database name to use in tests.
              Required when running tests without a Drupal installation that
              contains default database connection info in settings.php.
              Examples:
                mysql://username:password@localhost/databasename#table_prefix
                sqlite://localhost/relative/path/db.sqlite
                sqlite://localhost//absolute/path/db.sqlite

  --php       The absolute path to the PHP executable. Usually not needed.

  --concurrency [num]

              Run tests in parallel, up to [num] tests at a time.

  --all       Run all available tests.

  --module    Run all tests belonging to the specified module name.
              (e.g., 'node')

  --class     Run tests identified by specific class names, instead of group names.
              A specific test method can be added, for example,
              'Drupal\book\Tests\BookTest::testBookExport'.

  --file      Run tests identified by specific file names, instead of group names.
              Specify the path and the extension
              (i.e. 'core/modules/user/user.test').

  --types

              Runs just tests from the specified test type, for example
              run-tests.sh
              (i.e. --types "Simpletest,PHPUnit-Functional")

  --directory Run all tests found within the specified file directory.

  --xml       <path>

              If provided, test results will be written as xml files to this path.

  --color     Output text format results with color highlighting.

  --verbose   Output detailed assertion messages in addition to summary.

  --keep-results

              Keeps detailed assertion results (in the database) after tests
              have completed. By default, assertion results are cleared.

  --repeat    Number of times to repeat the test.

  --die-on-fail

              Exit test execution immediately upon any failed assertion. This
              allows to access the test site by changing settings.php to use the
              test database and configuration directories. Use in combination
              with --repeat for debugging random test failures.

  --browser   Deprecated, use --verbose instead. This enforces --keep-results and
              if you want to also view any pages rendered in the simpletest
              browser you need to add --verbose to the command line.

  --non-html  Removes escaping from output. Useful for reading results on the
              CLI.

  --suppress-deprecations

              Stops tests from failing if deprecation errors are triggered. If
              this is not set the value specified in the
              SYMFONY_DEPRECATIONS_HELPER environment variable, or the value
              specified in core/phpunit.xml (if it exists), or the default value
              will be used. The default is that any unexpected silenced
              deprecation error will fail tests.

  <test1>[,<test2>[,<test3> ...]]

              One or more tests to be run. By default, these are interpreted
              as the names of test groups as shown at
              admin/config/development/testing.
              These group names typically correspond to module names like "User"
              or "Profile" or "System", but there is also a group "Database".
              If --class is specified then these are interpreted as the names of
              specific test classes whose test methods will be run. Tests must
              be separated by commas. Ignored if --all is specified.

To run this script you will normally invoke it from the root directory of your
Drupal installation as the webserver user (differs per configuration), or root:

sudo -u [wwwrun|www-data|etc] php ./core/scripts/run-tests.sh
  --url http://example.com/ --all
sudo -u [wwwrun|www-data|etc] php ./core/scripts/run-tests.sh
  --url http://example.com/ --class "Drupal\block\Tests\BlockTest"

Without a preinstalled Drupal site and enabled Simpletest module, specify a
SQLite database pathname to create and the default database connection info to
use in tests:

sudo -u [wwwrun|www-data|etc] php ./core/scripts/run-tests.sh
  --sqlite /tmpfs/drupal/test.sqlite
  --dburl mysql://username:password@localhost/database
  --url http://example.com/ --all

Running a single simpletest class/method (Drupal 8)

To allow for faster iterations in your workflow, you may want to run a single test class or a single test method within a test class:

# Single test class
php core/scripts/run-tests.sh --verbose --url http://example.com/ --class "Drupal\module_name\Tests\TestClass"
# Single test method
php core/scripts/run-tests.sh --verbose --url http://example.com/ --class "Drupal\module_name\Tests\TestClass::testMethod"

Please, read the code. This page just scratches the surface as the test-running script mentioned here can be extended and re-used at will.

Friendly advice:

• Do make sure you run tests as web-server user.
• If you are running tests on a CLI from remote shell access, the screen(1) utility can be handy for long running test sessions.

Command-line test execution - Windows

Windows users do not have a ready-made .bat file that would make things click and go. However, the majority of the rules applies for Windows as well. All the options above work just the same. The easiest way to use this script is having a Bash environment. Using either Cygwin, or MSYS that comes with Git for Windows are both ample enough to use the Linux notes above.

To use this script with strictly Windows CMD, navigate to your drupal root, run the script using php while specifying the php installation you are using as follows:

php core\scripts\run-tests.sh --php C:\PHP5\php.EXE "My Test Group Here"

Keep in mind your version of PHP may be somewhere different. Example if you are using XAMP or Acquia Dev Desktop, your php executable may be somewhere in those installations.

You can also find out what PHP Executable you are currently using (for php >= 5.4) by typing:

php -r "echo PHP_BINARY;"

Lastly, there is an issue for the redundancy of having to re-specify the php executable in Windows at #2798377: Use PHP_BINARY Constant for run-tests.sh.

Running tests without an installed Drupal site (D8)

In Drupal 7, in order to run tests there must be an installed Drupal site with Simpletest module enabled. In Drupal 8 it is possible to run tests without enabling the module, and without installing the site at all, by using an sqlite database which is created on the fly.

Running php ./core/scripts/run-tests.sh --help displays the syntax. For example, you can run tests with this command (issued from document root):

sudo -u www-data php ./core/scripts/run-tests.sh --sqlite /tmpfs/test.sqlite --dburl mysql://username:password@localhost/database --url http://example.com/ --all

Breaking this command down a bit:

Use sudo to set the user to the user which will execute php (on Debian or Ubuntu, often 'www-data'). On a VM you may need to first make yourself root with 'sudo su root'.

The --sqlite flag tells the test runner to create a temporary database for the test. The name and the folder where you can store it can be anywhere in the file system provided the web server can write to it. After the test, the sqlite database can be cleaned for reuse using the --clean flag, or manually deleted.

Even if there is no installed site the --dburl flag is needed. In order to avoid errors the database should exist (though it can be completely empty) and the username (with database password if you are using one) should have access to it.

The --url flag is also needed and should be a working URL on your server, linked in your webserver configuration to the document root where the code to be tested is. The script will run without a working URL but many tests will fail. In order to set up a working URL when developing locally it is normally also necessary to create a line in the hosts file to generate a working URL: on Linux to use http://example.com/, you might add 127.0.0.1 example.com to /etc/hosts. On a local install of the Drupal testbot using the VM image described here (and probably some other VMs) you may also need to adjust the network interfaces to get the URL working, unless you place or link your code in a subdirectory of the site which is already set up at http://172.16.2.100/ .

The --all flag will run all tests. Replace this with the relevant tests using one of the flags described by the --help flag. To test one class, use for example --class "Drupal\action\Tests\ActionUninstallTest". Note that in same cases testing an individual class without an installed database will fail with 'table not found'.

Troubleshooting

If you see an error like:
simplexml_import_dom(): Invalid Nodetype to importsimplexml_import_dom(Object)

This usually means the test runner's browser is not working. Make sure your web server is running and you have passed a correct --url to the command.

If you have Drupal inside a virtual machine and access its web pages through a browser on your host machine, and you see an error like:
Test ExampleTests->setUp() failed: GET http://example.local/user returned 0 (0 bytes).
Even though you run your WebTestCases through the script or Drush, the virtual machine still needs to access 'example.local' (whereas for normal operation of the site it doesn't). So if you are pretty certain your code is not whitescreening at that point, try adding the domain name to your local hosts on the virtual machine.