test-run command fails using standard install profile

I looked at the gist script for a bit and tracked it down to the node_type_save() calls in standard_install(). There we eventually call comment_node_type_insert() which tries to create a field instance and eventually dies. No more time to debug just yet. I could keep going with this, but it would be lovely if someone else jumped in.

No answer yet, but got closer. The reason that the instance can't be created is because its field (comment_body) does not exist yet. The field was supposed to be created by the code below in comment.install but field_info_field('comment_body') returned TRUE and so the field creation code never gets called. I don't quite know whats going on.

if (!field_info_field('comment_body')) {
    $field = array(
      'field_name' => 'comment_body',
      'type' => 'text_long',
      'entity_types' => array('comment'),
    );
    field_create_field($field);
  }

?>

OK, got it.

This was yet another leakage between testing side and tested side. These are murderous bugs to track down.

Field API info gets statically cached the first time it is needed and then is held in memory until reset. It turns out that drush's final bootstrap phase, where we login the specified user or anon user, was triggerring an entity load which triggerred field_info_field(). Thus, we statically cached the fields as they exist on the testing side.

Then simpletest comes along and creates all new tables and generally thashed around and when comment_install() runs field_read_field() happily uses the static cache of what Fields exist where. Problem is, that cache refers to a different web site. The testing side has fields, but the tested does not yet have any. The tested side never gets its Fields because the Field API thinks they already exist. Thus, the instance creation fails. Everything worked when the testing side used minimal profile because that creates no fields.

The remedy in this case was for Drush to not attempt login and thus field info is not read too early. This is a somewhat brittle fix, since another commandfile could do something in a hook to early load Field info.

The real fix IMO is to not use Drupal for both your testing and tested sides. Its hopelessly confusing. This is Drupal simpletest's fatal flaw.

My gut is to use PHPUnit for #483940: Unit testing library. I think Drupal should move to PHPUnit as well. I know it is an immense project, but thats what we should explore at least.

Update - I changed the drush command so it only bootstraps to DATABASE. Now there is virtually no possibility of leakage between the two sites because the testing side has barely bootstrapped Drupal. Running all tests now, and at most are passing. UserPictureTestCase is not, for example. I consider these failing tests to be broken (its debateable).

Good question. PHPUnit is still the preferred platform for unit tests of drush itself as discussed at #483940: Unit testing library.

I think test-run is now a really good solution to the leakage problem. It might not be a complete solution but we are close. We need to fix any test failures that happen when run under test-run.

I should really blog about this stuff.

Correction. I thought we were bootstrapping to DATABASE but really we were doing FULL. I disabled the 'drupal dependencies=simpletest' line and then we did bootstrap to only to DATABASE. And then everything dies before you can run a test. From a new code comment in test-run:

// We'd like to not bootstrap at all but simpletest uses Drupal to discover test classes,
// cache the lists of tests, file_prepare_directory(), variable lookup like
// httpauth creds, copy pre-built registry table from testing side, etc.

In order to fix the leakage problem, someone has to test discovery and DrupalWebTestCase::setup so that they do not depend on a host drupal. It is plausible that the tests themselves don't need to change.