Automated JavaScript unit testing framework
dmitrig01 - March 22, 2008 - 19:19
| Project: | Drupal |
| Version: | 8.x-dev |
| Component: | javascript |
| Category: | feature request |
| Priority: | critical |
| Assigned: | boombatower |
| Status: | needs work |
| Issue tags: | rfaynovember |
Description
If we need full unit test coverage for code, I would assume this includes JavaScript.
I made a patch which has a light-weight unit-testing framework for JS.
Currently it has no UI, but you can run Drupal.test() through firebug.
Also, the checkPlain tests don't pass, and this is for good reason, because checkPlain is broken (yay tests!).
| Attachment | Size | Status | Test result | Operations |
|---|---|---|---|---|
| javascript_unit_testing.patch | 14.31 KB | Idle | Failed: Failed to apply patch. | View details | Re-test |
#1
The fix to the checkPlain bug is http://drupal.org/node/237577, and I have a new patch, which doesn't do stuff in drupal.js but moves tests over to tests.drupal.js.
I will file separate patches for more tests.
#2
Subscribe (thanks dmitri).
#3
I have found out a way to do functional testing. Now we need naming conventions for JS files
#4
This looks great! Subscribing.
#5
This looks terrific. Should we build upon the testing library of jquery - either testrunner or jqunit? Maybe you did already since they look similar.
#6
This is awesome. Thank you, Dmitri! (Subscribing. :))
#7
subscribing - thanks for the great work so far.
#8
subscribing...
#9
Is this something that we could invoke from within SimpleTests so that with one click on a button, we can run all tests, including the JS tests.
#10
Jquery has promoted its QUnit unit test framework to be an independant project. They *want* other projects to use this. We should consider it strongly. See http://docs.jquery.com/QUnit
#11
Thanks for taking the lead dmitrig01.
Yes, there are big potential advantages to using the QUnit framework, since it is the standard for jQuery, is under active external development, and has documentation and samples in place.
That said, (a) the QUnit documentation is sketchy so far, (b) Dmitri's version is coded more closely to our existing Drupal js codebase (c) there look to be some methods in this patch not available in QUnit, e.g., Drupal.tests.randomName.
Maybe rework the patch to build off of QUnit while adding missing methods (could also contribute as patches to QUnit).
@Dries, I guess we need to add a JS behaviour that calls JS tests? There are several significant challenges:
* As most (all?) of our JS behaviours depend on specific page content, we have the challenge of rendering that content and adding needed JS and CSS (possibly inline).
* Some JS behaviours will require specific user interaction and data (e.g., the ahah.js behaviors like file uploads).
Do we visit each page with a behaviour in sequence? Do we try to render on a single page all the minimal content needed by all behaviours? I guess the latter is more in line with what we do in other tests. We would require the simple tests to render the HTML needed for a test. Following the upload with AHAH example, this might be something like:
<?php
global $conf;
// Cache any existing variable.
$cached = $conf;
// Explicitly allow uploads with stories.
$conf['upload_story'] = TRUE;
// Build and render the upload form.
// Generate a dummy form.
$form = array();
// Upload's form_alter requires a #node property.
$node = new StdObj();
$node->type = 'story';
$form['#node'] = $node;
drupal_alter('form', $form, array(), 'story_node_form');
$form_state = array('submitted' => FALSE);
$form = form_builder('story_node_form', $form, $form_state);
// Rendering the form will ensure we have the appropriate AHAH JS/CSS loaded.
$output = drupal_render($form);
// Restore the previous $conf values.
$conf = $cached;
return $output;
?>
But we're still stuck with the problem of emulating user interaction. Iinitially at least, we might want to limit ourselves to higher level stuff that doesn't involve user actions.
#12
Hm, I think it might actually be a good idea to have them each on separate pages, and when defining a JS test in PHP (I was thinking of hook_jstest or something), you could have a callback that would get rendered into an iframe where all the assertions would happen.
#13
There must be a way we could get this working nicely in a SimpleTest test.
#14
Subscribing.
#15
The last submitted patch failed testing.
#16
See: #335122: Test clean HEAD after every commit and http://pastebin.ca/1258476
#17
Firing events is http://www.rakshith.net/blog/?p=35 and http://www.howtocreate.co.uk/tutorials/javascript/domevents apparently possible.
#18
Subscribing.
#19
subscribing
#20
subscribing
#21
I made a minor change to dmitri's patch so that Drupal.tests.testAttachBehaviors works with the new Drupal.behaviors.
#22
The last submitted patch failed testing.
#23
jQuery 1.3 came out with http://docs.jquery.com/Release:jQuery_1.3#Testing .
#24
John Resig is pushing the function call profiling awesomeness.
#25
I'm not sure if this is entirely relevant, but there are automated browser testing frameworks available that emulate user interaction. Selenium (http://seleniumhq.org/) [multiple languages] and Celerity (http://celerity.rubyforge.org/) [Ruby and can run headless] come to mind.
Essentially, you can code, "click on this link and wait for the page to load" or "click on this to see if the drop down box expands". Nifty interactions that are very high level.
It might be useful to run these sorts of tests outside of the simpletest framework - it would probably take a lot of work and code to integrate the two.
I've just started playing around with Celerity, so let me know if any of you guys are interested in pursuing this.
#26
subscribe
#27
@vaeiou selenium and celerity are bad because you have to install them in your browser.
#28
Now Resig is tackling the cross browser problem in quite an innovative way - crowdsourcing.
#29
i applied for the beta on behalf of drupal
#30
anyone up for this. is a big hole in our test suite. here is an article on qunit just to whet the appetite - http://highoncoding.com/Articles/570_Unit_Testing_JavaScript_Using_JQuer...
#31
Subscribe
BTW @dmitrig01 you are incorrect that you need to install Selenium in your browser - you only need to do that if you want to use the GUI test recorder - it is quite capable of running tests standalone (just extract the HTML and scripts anywhere in the webroot) or even driven by, and communicating with a server side script (with tests written in that scripts native language).
#33
This is a terribly important subject, as key failures are slipping through the system. A JS introduction to HEAD broke install.php completely the other day.
Let's have a BOF about this at Paris Drupalcon.
#34
Marking as non-critical in accordance with the issue priority guidelines.
#35
BOF on this subject today, Wednesday, 2 Sept, 18:00 at Drupalcon.
#36
...aaaand after discussion with webchick we'll mark this as "critical" again, at least until we have a resolution to the "What does Critical really mean" meta-issue. :)
#37
cwgordon7 and I work on integrating QUnit with SimpleTest.
#38
Doh
#39
Forgot the actual test suite.
EDIT: having upload issues.
#40
What works:
What needs to be done in future patches:
#41
This is an absolutely fantastic step forward for Drupal. With this patch, we now have the capability of at least doing Javascript unit testing, although we have a bit farther to go to provide functional testing.
As a review, what this patch does *right now*:
What it can do with a bit more work (very important work):
Here are a couple of screenshots showing what's been done - A new tab on the tests page for javascript tests, and a list of tests that have been run.
Thanks for the good work, cwgordon7 & boombatower.
#42
subscribing
#43
This looks awesome!!! Great, work guys!! :D
A couple of concerns from a quick glance-through:
1. I see verbiage to the effect that code in that new library is BSD. We need to check with Larry about that. I'm pretty sure it works to GPL BSD code, but we might need some fancy stuff.
2. When I click on "Javascript" tab, it initiates an action. This is weird. Every other admin page in Drupal requires an explicit action to trigger it.
#44
cwgordon and webchick asked me about the legal ramifications of including a 3rd party library for JS testing. The best source for that is here, section 2.1:
http://www.softwarefreedom.org/resources/2007/gpl-non-gpl-collaboration....
So if we include QUnit and make no changes to it, then we should include the file verbatim, including all existing copyright notices and information. The licenses on it are GPL-friendly so we're OK.
If we later add a CREDITS.txt file to core, we should include in it references to any of the copyright statements specified in the QUnit file.
#45
+++ modules/simpletest/simpletest.module 5 Sep 2009 15:10:51 -0000@@ -86,6 +111,28 @@
+function simpletest_library() {
+ $libraries = array();
+
We can remove that declaration, since we're using string keys.
+++ modules/simpletest/simpletest.pages.inc 5 Sep 2009 15:10:51 -0000@@ -194,6 +194,73 @@
/**
+ * User interface for running all JavaScript tests, which is triggered through
+ * the addition of an iframe which runs the JavaScript tests in the browser in
+ * a separate page callback.
+ */
...
+/**
+ * Runs all the JavaScript tests, this is triggered from qunit.php because if
+ * we don't have clean URLs enabled and use a Drupal path, QUnit tries to use
+ * the q GET parameter as a filter for itself.
+ */
...
+/**
+ * JavaScript callback to record JavaScript tests to the database. This is
+ * called once for each JavaScript test.
+ */
Missing PHPDoc summaries here.
+++ modules/simpletest/tests/drupal.test.js 1 Jan 1970 00:00:00 -0000@@ -0,0 +1,133 @@
+Drupal.tests.testT = {
...
+ test: function() {
Hmm... that additional "test" prefix makes no sense in JS.
Drupal.tests is the namespace already, "t" would be sufficient, because we additionally have the "test" method we try to execute.
+++ modules/simpletest/tests/drupal.test.js 1 Jan 1970 00:00:00 -0000@@ -0,0 +1,133 @@
+Drupal.tests.testBehaviors = {
...
+ setup: function() {
...
+ test: function() {
...
+ teardown: function() {
I'd recommend to move tearDown right after setUp - additionally, why don't we use the same capitalization as in PHP tests by the way?
+++ modules/simpletest/testrunner.js 1 Jan 1970 00:00:00 -0000@@ -0,0 +1,803 @@
+/*
Is this entirely copied? If not, then we want to fix the missing second asterisk here.
I'm on crack. Are you, too?
#46
Boombatower will post an updated patch from SVN shortly.
#47
Patch!
#48
The last submitted patch failed testing.
#49
Sorry about that, test included unrelated portions / old attempts, this one should be good though.
#50
Furthermore, here is some refactored JavaScript (that actually works this time), plus increased support for functional testing (I've written functional tests for jquery.once.js and collapse.js, but those belong in separate issues once this is committed -- the collapse.js test actually found a relatively nasty bug, too, which is even more case for having this in core).
I am strongly in favor of committing this sooner rather than later. The sooner this gets in, the more work can be done on JavaScript testing in core, which is critical both for our testing system as well as for our JavaScript code quality. Rfay gave this a solid review, and this patch additionally includes the support for functional testing that he was looking for. I would love to see this get in as soon as possible - I start school very soon, and would love to be able to do work on this while I still have an ample amount of free time.
#51
Chx pointed out https.php in IRC and asked form me to make qunit.php more like it. So here is an updated patch. :)
#52
And this one fixes having to click the "Run tests" button twice to make the tests run if they've already run. :)
#53
And here are collapse.test.js and jquery.once.test.js, which will go in separate patches later but are here for reference so that we can see how the API works for functional testing. Note that collapse.js.txt is currently broken due to the above mentioned core bug.
#54
Looks like we have some overlapping changes from those I made on plane. I'll try and merge these soon.
#55
Updating the initial list on #40:
What works:
What needs to be done in future patches:
I have cleaned up the backend code significantly. It has been split into three files:
The overlapping code/base code in simpletest.inc and the specific code for each in their respective files. The JS side now acts EXACTLY like the PHP side from the interface. The only things missing is detection of the JS tests properly...code is setup...but I haven't fleshed out the JS side of things. Right now the three JS tests are hardcoded in detection block.
Also note I have yet to update run-tests.sh as there are a couple of relevant changes. So bot will get a weird result.
More detailed list of changes:
#56
Subscriiiiiiiiiibe!
#57
The last submitted patch failed testing.
#58
Resubmit to test bot.
#59
The last submitted patch failed testing.
#60
I wasn't able to get the tests to run (see below), but code-wise I only saw minor issues. Let me know if I need to do something special to make it work, but ideally it should "just work" on install.
When I run the JS tests, the AJAX call to run the tests doesn't update the counter, and it runs batch?id=2&op=do over and over (I let it run for a long time). This was after updating to most recent HEAD, applying patch, dropping DB and then creating and installing, enabled admin menu and simpletest. "Provide verbose information when running tests" is enabled but I didn't change any other settings. The following was the AJAX response:
{"status":true,"percentage":"0","message":"Processed test 0 of 3 - \x3cem\x3eCheck plain\x3c\/em\x3e.\x3cdiv class=\"simpletest-pass\"\x3eOverall results: 0 passes, 0 fails, and 0 exceptions\x3c\/div\x3e\x3cdiv class=\"item-list\"\x3e\x3cul\x3e\x3cli class=\"first last\"\x3e\x3cdiv class=\"simpletest-pass\"\x3eCheck plain: 0 passes, 0 fails, and 0 exceptions\x3c\/div\x3e\x3c\/li\x3e\n\x3c\/ul\x3e\x3c\/div\x3e\x3ciframe src=\"http:\/\/d7.kernelpanic.local:8888\/modules\/simpletest\/qunit.php\" style=\"display: none;\"\x3e\x3c\/iframe\x3e"}-files[] = simpletest.install- just making sure this is intentional, not sure what the current state of the registry is regarding install files.The breadcrumb behavior is a little undesirable: Go to the settings form, then try to go up a level using the breadcrumb. It takes you to the PHP testing start page instead of letting you get to either. This isn't an issue that should hold up a commit though; it's so minor compared to the benefits.
Ensure that at least one tests is selected.singular?Ensure that the selected tests are valid and that at least one tests issame.If tests are not cached the goto simpletest/js/collect wherewording is strange.Copyright (c) 2008 John ResigOn this line in the testrunner.js file, there are UTF-8 encoding issues. A number of similar issues are further down the file.Can we just detect and avoid this situation? Infinite loops can be a big pain to debug, and any barrier to test creation is :(
+ // Note: never register a dependency on drupal.js, it is loaded+ // automatically anyway and will send this into an infinite loop.
test.js has a couple trailing whitespace issues. Also, need $Id$.
simpletest.qunit.tpl.php - I thought that having whitespace before the start tag for html was discouraged, but I could be wrong.
#61
subscribe
#62
Cleaned up some comment, made js test registration actually function (someone may want to look at js since behaviors auto-run so I had to put interesting logic at beginning).
Otherwise all is well, except batch API doesn't display proper numbers when doing js always as the POST requests are asynchronous and it grabs the results before they are fully recorded. I could make code to block, but it seems we would rather run tests faster...perhaps get rid of the results preview when running js tests?
Update run-tests.sh so bot should run.
Probably some other things that could be cleaned...but I think this patch is a big as we should make it (and as small).
As per #60....works fine on fresh install...with no changes other then patch...make sure you clear you caches. It removes and adds the entry in .info...(re-orders to alphabetical)
#63
I unfortunately confirmed the problems pointed out in #60 on a fresh install. The problem is that the iframe that runs the tests is being deleted and reinserted every time the batch API tries to update... so if you have too many tests, or just a slow computer, the iframe is deleted before the JavaScript can finish running the tests and posting the results. There were a few other minor quirks, but this is the only bug that's really preventing this from being committed as-is.
#64
I have attempted this, the batch api change is a peice of cake, but I can't get the proper blocking to work. It is simple logic with $_SESSION, but for some reason the batch API does not see the changes to $_SESSION...if I play with firebug or at random it works...leading me to believe there may be a concurrency issues that happens to screw with session.
I have spent 4-5 hours on this with no dice....taking a break for now.
#65
Random addition from legal: The CREDITS.txt file in core currently makes mention of jQuery as being a 3rd party lib. For consistency we should do the same if we're adding a 3rd party JS testing library, too. Just model on what is already there and include that in whatever the next patch is.
I have no technical comments at this time. :-)
#66
Ping. :) Would still love to see this.
#67
With the lack of a working batch API patch, I'd propose that we go back to the simpler approach in #52 - the batch API would be nice to have, but shouldn't be a blocker for the JavaScript testing framework. @boombatower: any thoughts?
#68
@boombatower: If you think we should proceed forward with your batch approach, please give full details of how to demonstrate the failure, and maybe we can get you some help. This is sooooo important. Thanks for all your work on this!
#69
Looking forward to putting some effort into this. Boombatower, please let me know exactly how to make this fail.
#70
Increase the time the js waits before posting results. Batch API will then run faster and continue to re-run the same test over and over since it never received results.
What you need is a flag when you got results or that your waiting..I've tried many variations, short of DB (although I did variable_set|get()). Possibly just make batch API wait longer.
This is extremely close as everything else works great: 1) test detection, 2) re-run, etc.
#71
Great! (subscribing)
#72
subscribe.
#73
subscribing
#74
Unfortunately, if this didn't get in by now it probably won't. Hopefully we can make it for D8.
#75
It would probably be good to start a javascript testing project in contrib (e.g. qunit), before trying to move it into core for Drupal 8. That way, we can have tests for Drupal 7 even if they're not in core, and it would be a great starting point for a JavaScript testing framework for Drupal 8.
Also, I'd like to note that I no longer think that using qunit is the best approach. While qunit is a wonderful automated JavaScript testing framework, my understanding is that it's really designed to test JavaScript independent of web content, not JavaScript that depends on PHP-driven web content (such as collapsible fieldsets, sticky table headers, tabledrag, and nearly every other major JavaScript implementation in Drupal). I think we should work out the details of a testing framework in contrib before coming in with a core patch.
#76
Agreed, that sounds like a good plan. And thanks for chiming in with your thoughts re: QUnit.
#77
Contrib is a great path forward.
We may want to return to the idea of using Selenium. Selenium is not a standalone approach, but it can test using many different browsers, which is critical for our javascript approach. We're far more likely to get real results. And selenium doesn't require anybody to know any javascript, so perhaps we can get more people involved in the test project.
Selenium would be more of a real-life test environment, so might be useful for a broader range of tests than just javascript/ajax related tests. There are places where the Simpletest environment, for all its glory and value, is too handicapped by using a virtual browser.
The upshot is that when we create a contrib project it's worth considering calling it something other than "jstest" or something, because it might have broader applicability.
#78
I really do not see the use of Selenium for Drupal, especially considering the extreme complexity it adds (actually running in all environments) (much less to add support for testing bot - having an X-server and/or windows running for IE and the like).
Drupal runs on top of jQuery and we should assume cross browser support (since they claim it), any cross-browser issues should be rare and/or css/html related. We just need a way to test all the ajax and drupal functionality built on top of jquery.
Having a simple script that runs in browser makes sense and would be much simpler to run on testbot.
I have spent a fair amount of time looking at Selenium and it was quite hard to actually get it going for what we would need for testbot and such (actually gave up). We have taken the DrupalWebTestCase approach for the majority of our functional testing needs...thus we just need to test the javascript components and form alterations.
We need to test one implementation of ajax autofill callback and the rest just need functional tests to ensure they spit out right data...etc.
#79
Selenium *is* a not-so-simple script that runs in the browser. It can also be extended to a far larger testing environment.
But I have *no* experience with it so cannot make the case. It was raised early in this issue (or a related one) and I know it's highly thought of in many circles for doing *one* of the things we need here, which is actual browser experience testing.
As far as javascript unit testing, I think we should use something like the path this was already taking. As far as AJAX forms and the like, we should keep our minds open and discuss the possibilities.