Exposing metadata to tests

@Damien:

What we are testing with functional tests is the operation of a complete product (Drupal), with its default theme and with some specific modules activated. We don't test the whole variability of parameters (what is we use a different theme? what if we enable all modules from the contrib repository? what if I play the guitar while Drupal is testing?).

Maybe in this phase we are only testing core modules, but Simpletest is not intended only for core, is it ? In my opinion it should be as powerful and extensible as Drupal is and can be used to test any related Drupal project.

Relying on the default theme's markup for assertions makes the testing system dependable of that theme. If we are testing a complete product, core tests must pass, no matter what theme or modules you are enabling.

I think this adds too much complexity, adds maintenance burden and simply change the way the product behave (you know, montoiring just changes the way the system behaves).

I do not agree with this. Where is the complexity ? In my opinion it reduces the complexity instead of adds to much complexity. The behavior of the system is not changed in any way. If a special user-agent is detected, it just respondes with a different template. The approach of detecting the user-agent is being used in core already. There is more complexity in the way that assertions are implemented now, often using tricky XPath expressions to get the data from the HTML output. Note, for example, a change in the class of an HTML element in the default theme, may breaks some assertions methods.

Anyway, if we need the real HTML output, we will have it in <content></content>.

I suggested in #243532: Catch notices, warnings, errors and fatal errors from the tested side to use HTTP headers to convey that information, but there are probably other ways to do this.

The approach of HTTP headers could work, but is not as extensible as XML is. Instead of injecting HTTP headers, isn't better to do this in the Drupal way?, just theming whatever we need to output.

How to test the patch
- Install Drupal using the string simpltest1 as the database prefix, or any string that matches /^simpletest\d+$/.
- Apply the patch.
- Now you guys can use for this Firefox with the user Agent Switcher Addon. Create a new User Agent, and use the same values as the database prefix. Then switch to this User Agent to simulate a simpletest request. When a request is made using the simpletest user agent, the output will be an XML as described above. Other browsers like Safari can be used with custom user agents too.

@catch: Suppose you are testing a custom module that in any way alters the user page (example.com/user) and the user form. DrupalWebTestCase::drupalLogin and drupalLogout base their assertions on finding the name and the password fields in a login form:

  $pass = $this->assertField('name', t('Username field found.'), t('Logout'));
  $pass = $pass && $this->assertField('pass', t('Password field found.'), t('Logout'));

So, when testing this module, if you try to logout a user with the default $this->drupalLogout() method will fail. It's true that you can overwrite this method in your test class, but this is a simple edge case, the real issue is that several assertions are based on the HTML output and relying on the default theme markup in some cases, what makes the testing system dependent of the theme, which isn't .

However, if you have the 'real' data that was used in the testing thread (the request to the testing site), tests will be more accurate and won't be basing their assertions on the HTML output, but the data received in the response, in a XML format, for example.

This way, the Logout method may be written this way:
// Load the user page, the idea being if you were properly logged out you should be seeing a login screen.
// Load the user page, the idea being if you were properly logged out, the user uid should 0.

  function drupalLogout() {
    // Make a request to the logout page.
    $this->drupalGet('logout');
    // Load the user page, the idea being if you were properly logged out, the user uid should 0
    $this->drupalGet('user');
    $this->_logged_in = ! $this->assertEqual((int)$this->xml->user->uid, 0, t('User logout successfully.')); 
}

In this case, the assertion is based on the data received, and not on the HTML output. $this->xml is a SimpleXMLElement object representing the XML data received.

Having the data will be useful for a lot of situations, where it can not be found by parsing the HTML output, or is hard to parse it, or is based on markup assumptions. Other good example is parsing the Drupal messages, errors, warnings, and status messages, etc... they can be obtained easily from the XML.

if you want to test the theme layer, well, no problem, you have the real HTML output in the <content></content> tag of the XML.

Here is a working patch implementing this idea. Note that the testing system is completely functional and the assertions continues working. Only modifications to the parsing method are required, so that the content now is in the <content></content> tag of the XML response. IF we use this technique, assertions can be improved later, using the data available.

the patch...

attachments working ?

@moshe: #145551: Enable loading and rendering into multiple formats (html, xml, json, atom, etc.) is a more general case, if that is the direction to go, then the rendering of the XML can be implemented that way. This issue is more related to the idea of making simpletest consume XML than to the rendering process. For now, this method can work, it requires fewer modifications to core and allow us to improve the testing system as described here. IMO, it's critical, the testing process could be easier if go this way.

Patch rerolled against HEAD.

Patch rerolled against HEAD.

Note that the testing system is completely functional after applying the patch. All the assertions continue working because only the parsing method is modified to get the content from the XML tag and not from the raw response.

@cwgordon7:

So override the functions. It's OOP

Yes, this can be an approach to deal with that issue, but this is not the main issue I am describing here.

We should be testing Drupal as it is, functionally - not some weird half-Drupal state of random xml data.

We will continue testing functionality, and will continue testing Drupal as it is. We only would be changing the way data is is received. Instead of parsing raw HTML data and basing assertions on data that is subject to change (for example, a simple change to garland theme may break a lot of assertions that are based on finding some markup on the HTML response). Anyway, if you still want to have access to the raw or 'real' HTML data, get if from the XML. I can see several advantages of doing so, we will be relying on correct and well structured data, and not in raw data that is difficult to parse and is state-less.

not some weird half-Drupal state of random xml data

Why random ?

While other format can be used to implement this idea, I think, XML is the more suitable due to its features like extensibility, readability, portability, security; and can be parsed in a similar way the system is parsing the XHTML output now.

Anyway, the idea is more general than using XML, it's more about exposing metadata to be consumed by tests and can be implemented using this approach later #145551: Enable loading and rendering into multiple formats (html, xml, json, atom, etc.)

Other possible approach may be the use of a simpletest specific theme, where theme functions may be implemented to expose some other data in the output. IMO, we should test Drupal with it's default output, and not with a specific theme, and moreover test each theme and the theme layer. So, a simpletest theme would be Drupal default output + other data exposed to tests (data that can not be parsed from the output or that is not accurate when parsing from the output).

So, if we want to test garland, then we run the garland test and see how are performing garland specific functionality. See #282540: Tests for themes

Also see this comment in this issue #279515: setting an installation profile for a test doesn't run hook_install_tasks()

@Damien:

I agree with you that we strongly need #243532: Catch notices, warnings, errors and fatal errors from the tested side as soon as possible, but this is the time to discuss and find complete and scalable solutions, and not particular solutions to a general problem: the absence of accurate info about the testing thread.

Again: we are testing Drupal core, not Drupal core with some fancy alterations.

I do not agree. Also see #3
IMHO, we are implementing a testing framework for Drupal, which can be seen as a content management framework or as a complete product (installed with the default profile or with any profile that includes multiple modules or themes), ...

Also you further may want to implement a test for a set of modules to test how they work together, or test an individual theme, ... which testing framework are you going to use? So, it's not only for core...

For example, think in this scenario: I maintain a drupal profile for an intranet, which includes a custom module to deal with LDAP. That module replaces the default Drupal login form with a specific form (requesting other information from the user, not just the username and password). I do not consider this a *fancy* alteration, but a real use case.

It must be able to use the testing framework to test such profiles, which enable certain modules and test how they work/interact with core, when multiple modules are enabled, and seen as a whole.

So, this issue is not only about the use of XML or a simpletest theme, it's about finding a way to expose some data to tests.

OK, if XML or a simpletest theme are not good aproaches, let's think in any other ;)

open for discussion.

in fact they only enable the minimum possible functionality in setUp() and clean it up when they're done - so an altered login form won't be taken into account during the core tests at all.

Won't be taken into account if you are running the default profile. Profile developers may want to run all tests using their custom built profile. As per #279515: setting an installation profile for a test doesn't run hook_install_tasks(), would be good to test alternate profiles to make sure it runs correctly.

Currently we have in setUp()

// Add the specified modules to the list of modules in the specified profile.
    $args = func_get_args();
    $modules = array_unique(array_merge(drupal_verify_profile($profile, 'en'), $args));
    drupal_install_modules($modules);

So, profile-installed modules + test case modules...

Also I don't think PHP warnings/notices etc. are special cases - they're pretty central to identifying bugs, and they're already displayed by default in development versions of Drupal - the only change is making the test framework pick them up.

Absolutely agreed. My point here is that we need to find a way to share such data that is available in the tested thread.