Unit testing library

The syntax behind phpt looks pretty nasty... is simpletest that much of a problem? No need to actually package it with drush...

I fail to see why simpletest is considered "too big" right off the bat... It's used in Drupal, and if it's too big for Drupal, well, I guess we could say Drupal is too big for drush and we still use drupal...

Let's not diverge from the library already used in the Drupal community, it's going to confuse things for everyone. By using simpletest, we make sure that a maximum number of people are already familiar with the testing framework and can contribute tests. Furthermore, we also avoid adding another dependency to drush as, arguably, simpletest is more likely to be installed...

I don't like shunit; it's too simple--or too complex, if you look at it the other way around. There's no substantial framework, and the shell scripts you need to write are inelegant.

I don't like doctest; placing tests inline in the php code itself would make the drush sources unwieldy and hard to scroll through.

I like phpt; seems about right in terms of size and capabilities.

Simpletest is a little more heavyweight than phpt, speaking only in terms of the code you need to write to get started. It has the advantage of being more similar to the Drupal "standard", which has merits. It seems If we used it, code that migrated from drush to drupal might take unit tests with it. But how likely is that? Rare, probably. I still think simpletest is imminently usable, although I like phpt more.

One thing to consider is that a large part of testing drush involves testing the command line parsing and bootstrapping code, not just individual functions. For that reason, something that actually calls drush from the shell would be beneficial. By this metric, shunit looks better -- but again, I'm not thrilled with it. Ideally, something that allowed different config files and environments to be put together would be ideal (use provision to create entire sites for testing?), but that's going down the "build it yourself" road, which is what would end up happening if you started using shunit. That would entail a phenomenal amount of work.

My opinions.

Hi

As the simpletest module is _the_ standard for unit/functional testing in Drupal, for me the ideal solution would be a part of that. It also seems like a bad idea to introduce a new API because of the added complexity. It would suck pretty hard writing a module and juggling two completely different testing frameworks around.

Simpletests assertions should cover pretty much what we need, and it already have a (messy) script for outputting reports to the shell. If there are specific needs we could extend the DrupalTestCase with a DrushTestCase class. Which obstacles are in the way for making that happen, and which command line specific features do we need to add?

All the best

I can understand why you would want to test drush itself without depending on drupal. Even though I still think [drupal] simpletest is the logical choice for doing this - PHPT would'nt be too bad.

But there is still a real need to have simpletest integration for the custom drush commands that your drupal module provides. Often you provide overlapping functionality in the backend and as drush commands and it makes a lot of sense that you can keep the testing of this functionality inside one testing framework.

Take care.

Bump!

I'm feeling the need for unit tests; it's taking too long to review changes I make. Reviewing this thread, I have some comments.

It would suck pretty hard writing a module and juggling two completely different testing frameworks around.

If you're writing a module with a drush interface, I think you're in pretty good shape. You can use (Drupal) SimpleTest to test all of your php functions. If the drush interface is thin enough, maybe you don't need to test it. If you do want to test the cli portion, though, I don't think that (Drupal) SimpleTest is the way to go. Even original simpletest or phpt is not ideal for testing from the cli.

I found shunit2, which is better than the first version quoted above, but still not great for cli testing. A long time ago (2001), I had some home-grown C++ test code that I used to build a cli-testing harness. Test cases were stored in text files (multiple tests per file) that looked something like this:

[Test "echo test"]$ echo test
test

[Test "cat test"]$ cat testdata.txt
This is the contents of the file 'testdata.txt', stored next to the unit test file

[Test "pipe test"]$ echo "This is a pipe test" | sed -e "s/pipe/working/"
This is a working test

Basically, [Test "description"]$ marked the beginning of a test. The stuff after the $ was exec'ed via bash. Everything from the next line down to the next test case was the expected output from the test. You could also add name/value pairs after the text description to pass parameters that set modes for the test (e.g. to expect a non-zero status result, or whatever). The main advantage of this format is that it was really easy to make tests; just type stuff into the shell, capture the output into an editor, and add "[Test]" labels in front of each command.

The cli harness was only about 200 lines[*], but it used a much larger C++ test framework that I wouldn't want to use here. I was considering porting this harness to php and layering it on top of (original) simpletest or phpt (probably the former). For testing drush php functions, you would instead make simpletest tests and call it with the drush simpletest command.

So, that would in fact be two different test frameworks, but I don't think that it would really be all that bad. The main downside is that this is largely a 'build your own' solution, but if the C++ version can be done in 200 lines, maybe the php port would be similarly short.

Comments? If I build it, do you think you'd use it?

[*] I measure lines of code after stripping comments, blank lines and any line with only a '}' or similar.

Just a quick note (as much to myself as anyone...) Simpletest has the following facility:

        function testEcho() {
            $shell = &new SimpleShell();
            $this->assertIdentical($shell->execute('echo Hello'), 0);
            $this->assertWantedPattern('/Hello/', $shell->getOutput());
        }

While it would not be terrible to use this pattern to test drush, I think there is some utility to the wrapper I mention in #11, which would certainly be easy to implement using the above. I'm working on this and will post something when it's ready.

I agree with the general sentiment above - I am happy with SimpleTest or something else, as long as we have a simple wrapper to run commands (then we check output strings, return code and site state etc).

I think having the ability for Drush to install Drupal core is pretty much required for testing any of the Drupal oriented commands to be repeatable and convenient. Right now even if we did have testing everyone would have to set up a dedicated site (probably several actually - one for each version) for us to test against, since testing against a real site could be destructive.

If we did have the ability to install Drupal however, we could bootstrap everything we need for the tests from scratch each time. We probably wouldn't need a working docroot, which simplifies things - we could just prompt for the mysql root password on the first run to create the databases and after that it would be fully automatic.

I would very much support plain old SimpleTest support. We don't need *Drupal's* SimpleTest to do what we need here, I agree that depending on Drupal is a bad idea.

I don't think we should go the way of another framework than what has been adopted by Drupal. I have yet to see a serious valid objection to plain old simple test here, just 'rumors' that simpletest was not the way to go for commandline testing, but who's talking about commandline here? Drush functions are functions like any other function and we don't absolutely need to test them by calling through the commandline: the test suite should call the functions themselves, without going through the commandline and report the results (yes, on the command line, but that's fairly trivial to bridge). Of course some commands will have to be tested through the commandline because they're designed to work that way (drush_invoke and friends come to mind) but there are a lot more functions (the bootstrap code, context stuff, etc) that don't interface with the console that needs testing that I don't think it's a strong enough argument against SimpleTest to warrant switching frameworks.

For the record, I never actually worked with simpletest and I am only summarily experienced with unit testing so my opinion should be taken with a grain of salt.

I am starting to work with SimpleTest, and it works well per #15. Testing the commandline can be done as shown in #13 without a wrapper, but I think that the wrapper I propose in #13 will be useful.

Drush's primary public API is the cli command interface, so it is my feeling that testing from the commandline is the most important. Individual functional testing per #15 is also important, and that of course should be done through simpletest. Note, however, that a lot of drush functions do drush_get_option or drush_set_option or use functions of Drupal, which implies a bootstrap and again brings us back to the commandline as the easiest way to test. Only functions with no side effects can easily be tested directly through the simpletest api, at least initially. Eventually we can write a system that will bootstrap to the appropriate level in setUp to allow fns with side effects to be tested one after another, but this is more complicated and arguably not a very accurate test anyway (since setUp might bootstrap differently than drush does; this technique is subject to code drift).

In any event, speculation at this phase is moot. My integration is coming along, and will allow you to mix the techniques of #11, #13 and direct php functional testing as discussed in #15. Please reserve judgement until you see the prototype.

My next hypothesis is that everything I am doing in original simpletest can also be done in Drupal simpletest. I will test this later. There is an advantage to this integration in that the Drupal package manager could check out HEAD and run a module's unit tests as part of a nightly build process. If it already does so, drush can benefit from hooking in to this process. If it doesn't, then perhaps we can help all of Drupal if we develop a testing system that can use drush to automatically create a test Drupal site for unit testing. That's certainly valuable.

One step at a time, though, and IMO, testing the cli is step number one.

Whoops, took a look at Drupal Simpletest, and as Moshe mentioned in #7, it has branched pretty far from original simpletest, and is missing everything in simpletest that would be used for cli testing. There's no point in trying to reconcile drush testing with Drupal Simpletest; you can call Drupal unit tests via the drush simpletest command.

Is simpletest really too big, though? I haven't decided on packaging yet. Simpletest is perhaps too large to check in to drush, but if it is not checked in, then we need to solve the problem of finding the simpletest dependency in a cross-platform way. Maybe check an environment variable, check well-known locations for different platforms, and download it (a-la table.inc) if it cannot be found?

(Edit: Fixed typo)

I'm stalled on this issue for now.

I've decided that I need to get Drupal Simpletest into my life before I make any decisions about how this should work for cli testing in drush. Drupal Simpletest can set up test sites during setup; should we use that instead of roll our own? Seems like it might be a good idea. If the (original) Simpletest cli-testing classes mentioned in #13 could be merged back into Drupal Simpletest, then maybe we could both have a nice drush-centric testing model and be integrated with the standard testing suite for Drupal.

But that remains to be seen. I'm going to have to look at it later.

Yes, I will probably roll back around to #16 by the time this is all done, but I do want to take a stroll down Drupal SimpleTest lane first and see if any of their setup work might be relevant to us.

sub

Any movement on this in the last month?

I have been reading up on the testing frameworks in this thread and I would love to have a crack at it if I am not stepping on anyone's toes

I for one would love to see suggestions and patches on this subject. I haven't spent any time on this since Januaray.

I think drush has definitely reached the point where it seriously needs a testing framework. The code base and usage cases have reached a point where two active maintainers can't hope to properly test every new piece of code that comes in. One example that comes immediately to mind is drush's support (or lack therof) for installation profiles. (cf. #806190: drush updb removes themes under /profiles from the system table, #831844: pm-download will happily create a second copy of a project in sites/all that already exists under an active profile directory, #827404: drush up drupal does not preserve custom profiles, #827888: Drush not picking up commands from modules located in an install profile in the help message?) These issues have hit us pretty hard, as we rely heavily upon drush to help us maintain a large infrastructure with many drupal sites. At this point, drush's code is equal in importance to drupal core for basically anyone that uses it...

That said, I'm not commenting just to complain, or add a useless +1. This is important enough to our organization that we're willing to devote time to create a patch. However, we'd need guidance on what the finished product should look like, in order not to waste a bunch of man-hours creating code that will ultimately be rejected. It looks like the stock simpletest library is the (reluctant) consensus. Is this true? If we don't want to include the simpletest library in drush, should there be a separate drush_tests project? Please advise.

I haven't read all the comments, but might I point out that if you expand on the existing Drupal Testing framework, primarily the unit test case, you will be able to take advantage of the automated testing system. Whereas if you go off and make your own you will not be able to use the automated testing system. Additionally I cannot see any reason why one would stray from the system used by the rest of Drupal, both for code/developer portability and usage of the automated tools that will only continue to be expanded in the future.

Another note, I question why any library needs to be committed to CVS. I think that was a mistake we made with the original SimpleTest.org library. Just like any jquery library it can easily be downloaded and dropped into any folder...or heck if you use the drupal version its just a regular module.

@boombatower: See my comments in #16, #17 and #18.

Hello!

Regarding #29 …I don't see how we're going to be able to test Drush _without_ a Drupal install; how else would we determine if running this or that function had the desired effect?

I realize that Dimpletest (Drupal SimpleTest module) is pretty web-centric at the moment, but if we do need those working Drupal installs available (and I can't imagine a way around that requirement), it would be nice to be able to utilize Dimpletest's pre-existing code for setting them up.

Thoughts?

Thanks,
Adam

Not sure how that is a mistake...I mean even if simpletest ran completely separate of drupal it should still generate a new "clean" drupal for testing and you would have still two drupals, but you would only be using one. At this point considering core already does it I would hardly consider that a solid argument against using it much less the fact that most people do not seem to have trouble understanding it.

Seems your quite against it, but I figured I'd go ahead and at least state my perspective and that of qa.d.o. I'll leave you all to it then.

Acrollet started a placeholder project for this called drush_unit. Just thought I'd put a link here for reference.

According to the PHPUnit site, "PHPUnit 3.5 requires PHP 5.2.7 (or later) but PHP 5.3.3 (or later) is highly recommended". What is the status of drush and 5.3? I know that 5.2 is required to run aegir but wasn't sure if this was drush related or not.

A couple of thoughts to add to the discussion:

- PHPUnit does look nice, it seems much more actively developed than simpletest. The main benefit simpletest had for core was the webrunner, which does not affect us.

- A possible side benefit of running PHPUnit, is that it might be possible to use it to wrap/format the core simpletests in some way, so that Drush's core test runner could provide output in a standard xUnit format without needing much extra work. This would make it much easier to set up continuous integration with Hudson and the like.

- One concern I have with a PHP based framework is that it might be susceptible to "environment pollution" from the calling environment. Considering that our environment adaptation stuff is one of our most fragile systems this could make it harder to test, relative to using (for example) shunit2 that start from the same place as the user. I am not saying that this means that we shouldn't use PHPUnit, just that we make sure we understand the environment effects of the different ways of calling drush commands (directly, and via the various exec-like mechanisms). One option to consider would be to have "drush-test" as a totally different entry point script than "drush".

Here are some scripts -- these correspond to the examples / diagrams in the Drush chapter of dgd7. These are not unit tests themselves, but they do run some drush commands in an automated way, which is cool an interesting in a more limited sense than full unit tests. There are limitations; you have to run the scripts in order, and some of them wait for user input. This might be useful starting point for generating test cases once a real test framework is in place.

Example output:

# ------------------------------------------------------------------ #
# Example 1-5: Selecting a site with --root and --uri
# ------------------------------------------------------------------ #

drush --root=$DIR/dgd7/web --uri=dgd7.org core-status "Site URI"
 Site URI   :  dgd7.org 

# ------------------------------------------------------------------ #

n.b. I am not advocating turning these into test cases by adding pass / fail messages, etc., but rather we should consider these as use cases for whatever test framework we choose to implement.

I think that roundup is a clear winner. Give me a bit of time and I will convert #40 into roundup tests -- maybe this weekend.

As I mentioned in #16, eventually we should also have some sort of php unit test library to do functional testing separately from cli testing, but cli testing is most important. Let's start here.

I think your php unit tests are great and should stick around forever, as they allow for functional testing in addition to cli testing. cli testing is more important in the short term, but long term I think we will want both. I think it's an open issue whether #43 or roundup will be better for cli testing.

1. It should be easy for 3rd parties to provide tests with their commandfiles; we should not test contrib commandfiles in drush core.
3. That sounds like a good idea to me. I'm planning to make some tests install an entire Drupal instance, per the scripts in #40; it would be good to be able to run just the tests that can be done without a test site. I don't know if it would be possible to write test suites that would run on just any Drupal site (without modifying said site) to allow quick testing w/out making a fresh test site every time. The more I think about it, the more I think that the DRUSH_BOOTSTRAP_DRUSH tests are going to be of limited utility, and we're going to have to take the hit of downloading and installing a Drupal test site that starts in a known good state for the most useful tests.

Here's a roundup version of the two tests you did, Moshe. Its pretty damn simple…

https://gist.github.com/777346

I much prefer the terseness of roundup, but it would also be nice to have something like phpunit that would integrate well with hudson.

Thanks. I haven't tried yet, but I expect that #40 should port to roundup pretty easily, since they are both just bash-based.

I had an idea a while ago that is still applicable today. I could write a wrapper around either roundup or phpunit to take tests in the form shown in #11 and run them through our engine of choice. Then we could have a bunch of terse cli tests where convenient, and fall back to hand-coded phpunit where added flexibility is required.

A thought, anyway. I suppose I'd have to code it up for any sort of real review. As I mentioned previously, I don't think it would be too many lines of code to set up.

Just googlestorming, I came across http://code.google.com/p/snaptest/ - I haven't tried to build some Drush tests in it yet, but at first sight it looks like a possible lightweight alternative to PHPunit (1.2M vs 8.3M). It is actively developed, and suggests it is xunit compatible.

Other than that I don't have any objections to PHPunit, given the examples above, it's not quite as terse as a shell based framework, but it is simple enough, and has the benefit that we can more easily use it for unit tests also (which some of the updatecode stuff could really use - all the x.y-a.b variations etc). We might want to write some kind of "environment mangler" extension/wrapper that we could use to create dummy environments to test more of the drush(.sh) and early bootstrap stuff in an isolated way. I think the xunit stuff is really important too, since I am sure at least a couple of us could easily host Hudson based continuous test systems, but if we have to do a lot of wiring it is going to get harder to maintain these and keep them consistent.

Yikes! We definitely do not want to test json with bash! If we use Roundup, it should be only for the cli tests. We should use php to test backend invoke, and to do functional testing.

I looked at snaptest, and it looked fine; however, I don't see that there is any advantage to it over phpnit other than size. If we just make the drush unit tests check for an (offer to) download phpunit when first run, then I don't think size is an issue.

At any rate, I've been looking at this for the past several days and have minor dissatisfactions about this and that, but have not quite reached any conclusions yet. I think that we should continue with php unit, and look for some easy way to write a wrapper for it to make cli tests easier. That's the direction I'm going to continue my investigations in, anyway.

I like #46, but think that we need to move the configuration items from phpunit.xml.dist into some drushrc.php file (or generate phpunit.xml from there), so that we don't have one more place to keep configuration options. It's a bother that you simply must have your database settings pre-configured before running tests, but that's the way it's got to be.

Regarding phpunit.xml, we could have a drush command called through to the custom test runner, which generates phpunit.xml. Run that just once, and thereafter you can use the standard phpunit with your IDE or whatever.

There are multiple issues involved with making tests easier. There is setup, as Owen mentions in #50. There is comparison methods, to test to see if the result is good. Some commands, like pm-releases, are hard to write tests for because the results will change as the contents of drupal.org evolve. For testing sql-sync & c., some tests will require checking to see if two Drupal sites are the same. I think that I can fix #758852: drush sql-hash to work with mysql to serve this purpose. Finally, it is nicer to say drush pm-download devel --destination=$detination than $this->drush('pm-download', array('devel'), array('destination' => $destination));. Better to use the later than to not write tests at all, of course, but once this is up and running, we might be able to make things a little more convenient. Maybe extend Drush_DrupalTestCase, load a script into a PHP variable via a HEREDOC, and pass it to a custom interpreter that will call backend invoke repeatedly, and do the appropriate assertions on the result. Then we could still run tests via php unit, but it would be really fast to generate new tests in this form. When I was using the test framework described in #11, I could just set PS1, type stuff in bash, and capture the output in a test file. That might work here too.

No, we don't need to test bash $PATH handling; I'd say that any time we exec drush, we should use the full path.

I am not sure I understand the issue with exec() here... but it seems that PHPUnit could just call drush with --backend and read the json output, if necessary (no?).

That said - what's the planned timeline to merge the unit tests in Drush 5? I would like to see this land so that we can start doing our own unit testing in provision/aegir - or should we not depend on Drush's API for that?

I noticed drush_make uses its own testing procedures, not sure this could be considered unit tests however... http://drupalcode.org/project/drush_make.git/blob_plain/refs/heads/6.x-2...