Unit Testing Plan

When I test module simple test on ubuntu, The module is not active. Who know why?

Here's a better idea for the simpletest return thing -

function simpletest_change_return($function, $value) {
  runkit_function_redefine($function, '', 'return '. var_export($value, 1) .';');
}

Very simple, eh?
Just call simpletest_change_return('drupal_get_form', array('#type' => 'form')); and drupal_get_form(); will return array('#type'=>'form').

Not exactly.
If we figure out how to run one test per page load, then it would work (the changes are per-page-load).

We had a long talk (we being boombatower, Crell, catch, myself, and probably other people I forget) about how adding a runkit requirement is going to make it so that only 0.000001% of people can run unit tests, which would basically defeat the purpose. And a better approach is refactoring of these monster functions that call functions that call functions as Crell did for the registry patch. See before/after here for a guide: http://pastebin.ca/961193

I agree with that sentiment - runkit may have a place for advanced tests for critical and complicated functions, but it shouldn't be required for the majority of tests.

+1 for refactoring our code to make it possible to do unit testing rather than using runkit to go around the problem

Subscribing.

One thing that hasn't been mentioned much here is "private" vs. public functions. Typically in test-driven development, only public methods are tested since the internal working of a class (private methods) do not need to be tested on their own (given that they are not called ouside of the class itself), and are subject to refactoring. NOT creating tests for private methods (indicated by a leading _) would cut down on the number of tests that need to be created as well as make it simpler to refactor the public methods without rewriting tons of tests.

In addition, I've seen that some people have suggested using runkit or some other methodology to create mocks for what would be considered the "private" functions. I think that this would be a mistake because it would require anyone modifying a private method's parameters or returns to find all of the mocks and modify them as well. This is prone to errors and it would be easier if the errors were pointed out by test failures when a private method is changed.

What does everyone think of this? Since we are using procedural code, there is no enforcement of private vs. public methods, so that could be an issue

I agree that public functions will still call other public functions, but by using mocks like runkit, you can inadvertently hide the dependencies. For example, if foo() calls bar() and I refactor bar() to return a different value or accept different parameters, the tests for foo() will pass, but in actuality, foo() will be broken.

I do agree that there will be some need for mocks. A good example is db_query(), since its return value and parameters are very unlikely to change and it requires that the database be set up, have the correct data, etc. I am just saying that using it to set up an environment for tests can be dangerous in my opinion.

Also, I am very interested in helping with the unit testing project in general. Boombatower, you seem to be one of the main contributors, especially recently, so if you know of some specific things that you know need to be done, I'd be happy to help.

OK, here is a modified proposal (I've read through a lot, but please let me know if I missed anything that has been discussed...I'm new here)

File Structure

One class per file, once class per core file. block.admin.inc -> block.admin.inc.test

File Naming

Each file is named after the file it tests

Example:
block.admin.inc -> block.admin.inc.test
block.module -> block.module.test

TestCase Class Naming

The classes will then be named in a similar fashion.
block.admin.inc -> BlockAdminIncTestCase
block.module -> BlockModuleTestCase

Grouping

Tests should be grouped by module for unit tests. All tests relating to a single module must pass before it is ready to be committed.

Example:
block.admin.inc -> block.admin.inc.test
block.module -> block.module.test

These would be in a group called Block. Each test class could be run individually or the group could be run as a whole.

Test Structure

Each public function should have a single test assigned to it. Private functions (denoted by a leading _) do NOT have tests assigned to them. We should make the note to module developers that they are strongly discouraged from using private core functions in their modules because they are not tested and thus subject to refactoring and change.

Test Naming

Test naming would be camelCased to comply with most OOP conventions and SimpleTest's own naming convetion.

Example:
db_query -> testDbQuery()

Mock Functions

This one is a little hairy because of the runkit problem mentioned above. I don't see a way around using runkit to generate the mocks that are required to get tests working consistently without a consistent database/file system. However, I propose that no mocks be generated for the module/file that is currently being tested and that it be limited to the current test case (method).

If there is a case where a mock or several mocks might be useful across several methods, a private method can be created and called in each test case that it is needed.

I don't know of a way to redefine functions besides using runkit, so we might have to require that people have it installed to run certain tests. This is especially the case for tests that deal with the database and file system.

Backwards Compatibility

An implication of standardized unit testing is that it forces much more emphasis on a stable public API. If we start changing the parameters and return of public functions, tests using those functions in other modules will start breaking. I have seen other frameworks put fixes (sometimes temporary) when the public API does have to change. This is not really an issue for this proposal, but it is something to think about because having these tests will make it twice as much work to change the public API.

Automation

Boombatower has created a module that automates the majority of the above mentioned that is possible to automate. He will update it once a convention has been decided on. He will then generate all the test cases and commit them. This will provide a consistent base for tests to be filled in.

That's it for now. Any thoughts on these issues or others?

I'm definitely in favor of refactoring code where possible, and limit mock objects to data sources external to php (filesystem, db, remote data, etc). Besides the fact that most testers won't have the runkit extension installed, it's also a great occasion to improve the general readability of Drupal's code.

I found a way to redefine functions without runkit. This will require big but simple changes to Drupal core. Let's call the current module and include files "bundles" just to make it simpler to describe what I want to do here. In CVS, Drupal will have a lot of includes -- one function per include. We supply a simple script that builds the aformentioned bundles. So the shipping tarball will have the same modules and includes -- . We run this script before packing Drupal. Also, anyone checking out CVS can run this sript. Once we have the registry in it's easy to patch it to support loading of a bundle of includes instead of just one file. So when you want to unit test function X then you change the filepath of functions called by this function in the registry table to the testing version of these functions and then boot Drupal -- so we will need to run unit tests through the browser as well.

@chx - this sounds like a very interesting idea, but I think the practical aspects need a little more fleshing out.

For example, every core install should be able to run the unit tests, right? So we'd want to ship in the tarball all the individual function files as well as the "compiled" files?

In this case, I guess development actually happens with diffs against these 1000's of individual function files? Is that practical?

How about the opposite path - a script breaks down the current module and include files into individual function files that get saved within the testing directory somewhere. This allows us to autogenerate predictable file names, and to not worry if a function is moved between files. Then the registry is tweaked to point to this collection of individual files during the tests, and as you propose above, you can substitute individual functions as needed.

Opposite path -- very good idea, thanks. We might want to ask the registry to do the split, actually -- it has parsing already built in anyways, the necessary code to write them out is a few lines of code. So much nicer than doing it in CVS. Thanks , thanks!

About "every core install should be able to run the unit tests" -- no, not at all. This is not a goal. As many as possible, that is a goal. For eg. if they don't have curl or simplexml extensions, they won't be able to run browser based tests (though unit tests might be able to run from a command line Drupal but then you need php-cli).

@chx - Yes, you're correct- I should rather have said that my understanding is that Drupal core should ship with all the code and files necessary to do all the tests, though not all may be runable depending on the PHP version and extensions.

Mocking functions
http://www.phppatterns.com/docs/develop/simpletest_mock_functions
Uses runkit of course.

Concept of mocking is incredibly handy in and of itself, but bonus feature, it manages the restoring of functions.

I investigated package some, all of it's own unit tests passed for me except something about wildcard. There is not a lot
to the library, and it's not a real release. I'd be happy to contact original owner and inquire about it's status if people
think this would be useful.

subscribing

Hm... would it be an option to run the function under a different name? Like,

$replacements = array(
  'replaced_function' => $replacement_function,
  ...more of those...
);
$result = BlockAdminIncTestCase->invoke($function, $args, $replacements);

invoke() then gets the string of the function in question (by parsing the associated source file), tokenizes the function with token_get_all(), replaces the function names in $replacements and also replaces the own function name with something else (say, "simpletest_generated_$function").

Then the parsed tokens are reassembled, written to a temporary include file, that file is included, the renamed/rewritten function is called, and the include file is deleted again.

Did I miss something that would prevent this from working?

subscribing

@jpetso - I think you are missing the point. The idea is to be able to substitute the functions call by whatever function is being tested.

Ok, I'm not surprised that this is the case. Just thought that if the use case is narrow enough, the above approach might work too, but if it doesn't, that's cool with me. Thanks for the clarification.

What exactly is the state of unit testing now? The tutorial page mentions DrupalUnitTestCase, and there appears to be a file implementing it in CVS, but it's not in the current Drupal 6 tar files. Is it simply something that hasn't been put into the D6 branch? I haven't studied it in detail, but it doesn't seem like the sort of thing that would have significant D6/D7 dependencies, and the tutorial page does seem to focus on D6.

Or is the best approach to use SimpleTest directly, without the SimpleTest module?