Posted by dmitrig01 on March 22, 2008 at 7:19pm
| Project: | Drupal core |
| Version: | 8.x-dev |
| Component: | javascript |
| Category: | feature request |
| Priority: | major |
| Assigned: | boombatower |
| Status: | needs work |
| Issue tags: | jQuery, QUnit, rfaynovember, Rob Loach's Wishlist, Simpletest |
Issue Summary
If we need full unit test coverage for code, I would assume this includes JavaScript.
I made a patch which has a light-weight unit-testing framework for JS.
Currently it has no UI, but you can run Drupal.test() through firebug.
Also, the checkPlain tests don't pass, and this is for good reason, because checkPlain is broken (yay tests!).
| Attachment | Size | Status | Test result | Operations |
|---|---|---|---|---|
| javascript_unit_testing.patch | 14.31 KB | Idle | Failed: Failed to apply patch. | View details | Re-test |
Comments
#1
The fix to the checkPlain bug is http://drupal.org/node/237577, and I have a new patch, which doesn't do stuff in drupal.js but moves tests over to tests.drupal.js.
I will file separate patches for more tests.
#2
Subscribe (thanks dmitri).
#3
I have found out a way to do functional testing. Now we need naming conventions for JS files
#4
This looks great! Subscribing.
#5
This looks terrific. Should we build upon the testing library of jquery - either testrunner or jqunit? Maybe you did already since they look similar.
#6
This is awesome. Thank you, Dmitri! (Subscribing. :))
#7
subscribing - thanks for the great work so far.
#8
subscribing...
#9
Is this something that we could invoke from within SimpleTests so that with one click on a button, we can run all tests, including the JS tests.
#10
Jquery has promoted its QUnit unit test framework to be an independant project. They *want* other projects to use this. We should consider it strongly. See http://docs.jquery.com/QUnit
#11
Thanks for taking the lead dmitrig01.
Yes, there are big potential advantages to using the QUnit framework, since it is the standard for jQuery, is under active external development, and has documentation and samples in place.
That said, (a) the QUnit documentation is sketchy so far, (b) Dmitri's version is coded more closely to our existing Drupal js codebase (c) there look to be some methods in this patch not available in QUnit, e.g., Drupal.tests.randomName.
Maybe rework the patch to build off of QUnit while adding missing methods (could also contribute as patches to QUnit).
@Dries, I guess we need to add a JS behaviour that calls JS tests? There are several significant challenges:
* As most (all?) of our JS behaviours depend on specific page content, we have the challenge of rendering that content and adding needed JS and CSS (possibly inline).
* Some JS behaviours will require specific user interaction and data (e.g., the ahah.js behaviors like file uploads).
Do we visit each page with a behaviour in sequence? Do we try to render on a single page all the minimal content needed by all behaviours? I guess the latter is more in line with what we do in other tests. We would require the simple tests to render the HTML needed for a test. Following the upload with AHAH example, this might be something like:
<?php
global $conf;
// Cache any existing variable.
$cached = $conf;
// Explicitly allow uploads with stories.
$conf['upload_story'] = TRUE;
// Build and render the upload form.
// Generate a dummy form.
$form = array();
// Upload's form_alter requires a #node property.
$node = new StdObj();
$node->type = 'story';
$form['#node'] = $node;
drupal_alter('form', $form, array(), 'story_node_form');
$form_state = array('submitted' => FALSE);
$form = form_builder('story_node_form', $form, $form_state);
// Rendering the form will ensure we have the appropriate AHAH JS/CSS loaded.
$output = drupal_render($form);
// Restore the previous $conf values.
$conf = $cached;
return $output;
?>
But we're still stuck with the problem of emulating user interaction. Iinitially at least, we might want to limit ourselves to higher level stuff that doesn't involve user actions.
#12
Hm, I think it might actually be a good idea to have them each on separate pages, and when defining a JS test in PHP (I was thinking of hook_jstest or something), you could have a callback that would get rendered into an iframe where all the assertions would happen.
#13
There must be a way we could get this working nicely in a SimpleTest test.
#14
Subscribing.
#15
The last submitted patch failed testing.
#16
See: #335122: Test clean HEAD after every commit and http://pastebin.ca/1258476
#17
Firing events is http://www.rakshith.net/blog/?p=35 and http://www.howtocreate.co.uk/tutorials/javascript/domevents apparently possible.
#18
Subscribing.
#19
subscribing
#20
subscribing
#21
I made a minor change to dmitri's patch so that Drupal.tests.testAttachBehaviors works with the new Drupal.behaviors.
#22
The last submitted patch failed testing.
#23
jQuery 1.3 came out with http://docs.jquery.com/Release:jQuery_1.3#Testing .
#24
John Resig is pushing the function call profiling awesomeness.
#25
I'm not sure if this is entirely relevant, but there are automated browser testing frameworks available that emulate user interaction. Selenium (http://seleniumhq.org/) [multiple languages] and Celerity (http://celerity.rubyforge.org/) [Ruby and can run headless] come to mind.
Essentially, you can code, "click on this link and wait for the page to load" or "click on this to see if the drop down box expands". Nifty interactions that are very high level.
It might be useful to run these sorts of tests outside of the simpletest framework - it would probably take a lot of work and code to integrate the two.
I've just started playing around with Celerity, so let me know if any of you guys are interested in pursuing this.
#26
subscribe
#27
@vaeiou selenium and celerity are bad because you have to install them in your browser.
#28
Now Resig is tackling the cross browser problem in quite an innovative way - crowdsourcing.
#29
i applied for the beta on behalf of drupal
#30
anyone up for this. is a big hole in our test suite. here is an article on qunit just to whet the appetite - http://highoncoding.com/Articles/570_Unit_Testing_JavaScript_Using_JQuer...
#31
Subscribe
BTW @dmitrig01 you are incorrect that you need to install Selenium in your browser - you only need to do that if you want to use the GUI test recorder - it is quite capable of running tests standalone (just extract the HTML and scripts anywhere in the webroot) or even driven by, and communicating with a server side script (with tests written in that scripts native language).
#33
This is a terribly important subject, as key failures are slipping through the system. A JS introduction to HEAD broke install.php completely the other day.
Let's have a BOF about this at Paris Drupalcon.
#34
Marking as non-critical in accordance with the issue priority guidelines.
#35
BOF on this subject today, Wednesday, 2 Sept, 18:00 at Drupalcon.
#36
...aaaand after discussion with webchick we'll mark this as "critical" again, at least until we have a resolution to the "What does Critical really mean" meta-issue. :)
#37
cwgordon7 and I work on integrating QUnit with SimpleTest.
#38
Doh
#39
Forgot the actual test suite.
EDIT: having upload issues.
#40
What works:
What needs to be done in future patches:
#41
This is an absolutely fantastic step forward for Drupal. With this patch, we now have the capability of at least doing Javascript unit testing, although we have a bit farther to go to provide functional testing.
As a review, what this patch does *right now*:
What it can do with a bit more work (very important work):
Here are a couple of screenshots showing what's been done - A new tab on the tests page for javascript tests, and a list of tests that have been run.
Thanks for the good work, cwgordon7 & boombatower.
#42
subscribing
#43
This looks awesome!!! Great, work guys!! :D
A couple of concerns from a quick glance-through:
1. I see verbiage to the effect that code in that new library is BSD. We need to check with Larry about that. I'm pretty sure it works to GPL BSD code, but we might need some fancy stuff.
2. When I click on "Javascript" tab, it initiates an action. This is weird. Every other admin page in Drupal requires an explicit action to trigger it.
#44
cwgordon and webchick asked me about the legal ramifications of including a 3rd party library for JS testing. The best source for that is here, section 2.1:
http://www.softwarefreedom.org/resources/2007/gpl-non-gpl-collaboration....
So if we include QUnit and make no changes to it, then we should include the file verbatim, including all existing copyright notices and information. The licenses on it are GPL-friendly so we're OK.
If we later add a CREDITS.txt file to core, we should include in it references to any of the copyright statements specified in the QUnit file.
#45
+++ modules/simpletest/simpletest.module 5 Sep 2009 15:10:51 -0000@@ -86,6 +111,28 @@
+function simpletest_library() {
+ $libraries = array();
+
We can remove that declaration, since we're using string keys.
+++ modules/simpletest/simpletest.pages.inc 5 Sep 2009 15:10:51 -0000@@ -194,6 +194,73 @@
/**
+ * User interface for running all JavaScript tests, which is triggered through
+ * the addition of an iframe which runs the JavaScript tests in the browser in
+ * a separate page callback.
+ */
...
+/**
+ * Runs all the JavaScript tests, this is triggered from qunit.php because if
+ * we don't have clean URLs enabled and use a Drupal path, QUnit tries to use
+ * the q GET parameter as a filter for itself.
+ */
...
+/**
+ * JavaScript callback to record JavaScript tests to the database. This is
+ * called once for each JavaScript test.
+ */
Missing PHPDoc summaries here.
+++ modules/simpletest/tests/drupal.test.js 1 Jan 1970 00:00:00 -0000@@ -0,0 +1,133 @@
+Drupal.tests.testT = {
...
+ test: function() {
Hmm... that additional "test" prefix makes no sense in JS.
Drupal.tests is the namespace already, "t" would be sufficient, because we additionally have the "test" method we try to execute.
+++ modules/simpletest/tests/drupal.test.js 1 Jan 1970 00:00:00 -0000@@ -0,0 +1,133 @@
+Drupal.tests.testBehaviors = {
...
+ setup: function() {
...
+ test: function() {
...
+ teardown: function() {
I'd recommend to move tearDown right after setUp - additionally, why don't we use the same capitalization as in PHP tests by the way?
+++ modules/simpletest/testrunner.js 1 Jan 1970 00:00:00 -0000@@ -0,0 +1,803 @@
+/*
Is this entirely copied? If not, then we want to fix the missing second asterisk here.
I'm on crack. Are you, too?
#46
Boombatower will post an updated patch from SVN shortly.
#47
Patch!
#48
The last submitted patch failed testing.
#49
Sorry about that, test included unrelated portions / old attempts, this one should be good though.
#50
Furthermore, here is some refactored JavaScript (that actually works this time), plus increased support for functional testing (I've written functional tests for jquery.once.js and collapse.js, but those belong in separate issues once this is committed -- the collapse.js test actually found a relatively nasty bug, too, which is even more case for having this in core).
I am strongly in favor of committing this sooner rather than later. The sooner this gets in, the more work can be done on JavaScript testing in core, which is critical both for our testing system as well as for our JavaScript code quality. Rfay gave this a solid review, and this patch additionally includes the support for functional testing that he was looking for. I would love to see this get in as soon as possible - I start school very soon, and would love to be able to do work on this while I still have an ample amount of free time.
#51
Chx pointed out https.php in IRC and asked form me to make qunit.php more like it. So here is an updated patch. :)
#52
And this one fixes having to click the "Run tests" button twice to make the tests run if they've already run. :)
#53
And here are collapse.test.js and jquery.once.test.js, which will go in separate patches later but are here for reference so that we can see how the API works for functional testing. Note that collapse.js.txt is currently broken due to the above mentioned core bug.
#54
Looks like we have some overlapping changes from those I made on plane. I'll try and merge these soon.
#55
Updating the initial list on #40:
What works:
What needs to be done in future patches:
I have cleaned up the backend code significantly. It has been split into three files:
The overlapping code/base code in simpletest.inc and the specific code for each in their respective files. The JS side now acts EXACTLY like the PHP side from the interface. The only things missing is detection of the JS tests properly...code is setup...but I haven't fleshed out the JS side of things. Right now the three JS tests are hardcoded in detection block.
Also note I have yet to update run-tests.sh as there are a couple of relevant changes. So bot will get a weird result.
More detailed list of changes:
#56
Subscriiiiiiiiiibe!
#57
The last submitted patch failed testing.
#58
Resubmit to test bot.
#59
The last submitted patch failed testing.
#60
I wasn't able to get the tests to run (see below), but code-wise I only saw minor issues. Let me know if I need to do something special to make it work, but ideally it should "just work" on install.
When I run the JS tests, the AJAX call to run the tests doesn't update the counter, and it runs batch?id=2&op=do over and over (I let it run for a long time). This was after updating to most recent HEAD, applying patch, dropping DB and then creating and installing, enabled admin menu and simpletest. "Provide verbose information when running tests" is enabled but I didn't change any other settings. The following was the AJAX response:
{"status":true,"percentage":"0","message":"Processed test 0 of 3 - \x3cem\x3eCheck plain\x3c\/em\x3e.\x3cdiv class=\"simpletest-pass\"\x3eOverall results: 0 passes, 0 fails, and 0 exceptions\x3c\/div\x3e\x3cdiv class=\"item-list\"\x3e\x3cul\x3e\x3cli class=\"first last\"\x3e\x3cdiv class=\"simpletest-pass\"\x3eCheck plain: 0 passes, 0 fails, and 0 exceptions\x3c\/div\x3e\x3c\/li\x3e\n\x3c\/ul\x3e\x3c\/div\x3e\x3ciframe src=\"http:\/\/d7.kernelpanic.local:8888\/modules\/simpletest\/qunit.php\" style=\"display: none;\"\x3e\x3c\/iframe\x3e"}-files[] = simpletest.install- just making sure this is intentional, not sure what the current state of the registry is regarding install files.The breadcrumb behavior is a little undesirable: Go to the settings form, then try to go up a level using the breadcrumb. It takes you to the PHP testing start page instead of letting you get to either. This isn't an issue that should hold up a commit though; it's so minor compared to the benefits.
Ensure that at least one tests is selected.singular?Ensure that the selected tests are valid and that at least one tests issame.If tests are not cached the goto simpletest/js/collect wherewording is strange.Copyright (c) 2008 John ResigOn this line in the testrunner.js file, there are UTF-8 encoding issues. A number of similar issues are further down the file.Can we just detect and avoid this situation? Infinite loops can be a big pain to debug, and any barrier to test creation is :(
+ // Note: never register a dependency on drupal.js, it is loaded+ // automatically anyway and will send this into an infinite loop.
test.js has a couple trailing whitespace issues. Also, need $Id$.
simpletest.qunit.tpl.php - I thought that having whitespace before the start tag for html was discouraged, but I could be wrong.
#61
subscribe
#62
Cleaned up some comment, made js test registration actually function (someone may want to look at js since behaviors auto-run so I had to put interesting logic at beginning).
Otherwise all is well, except batch API doesn't display proper numbers when doing js always as the POST requests are asynchronous and it grabs the results before they are fully recorded. I could make code to block, but it seems we would rather run tests faster...perhaps get rid of the results preview when running js tests?
Update run-tests.sh so bot should run.
Probably some other things that could be cleaned...but I think this patch is a big as we should make it (and as small).
As per #60....works fine on fresh install...with no changes other then patch...make sure you clear you caches. It removes and adds the entry in .info...(re-orders to alphabetical)
#63
I unfortunately confirmed the problems pointed out in #60 on a fresh install. The problem is that the iframe that runs the tests is being deleted and reinserted every time the batch API tries to update... so if you have too many tests, or just a slow computer, the iframe is deleted before the JavaScript can finish running the tests and posting the results. There were a few other minor quirks, but this is the only bug that's really preventing this from being committed as-is.
#64
I have attempted this, the batch api change is a peice of cake, but I can't get the proper blocking to work. It is simple logic with $_SESSION, but for some reason the batch API does not see the changes to $_SESSION...if I play with firebug or at random it works...leading me to believe there may be a concurrency issues that happens to screw with session.
I have spent 4-5 hours on this with no dice....taking a break for now.
#65
Random addition from legal: The CREDITS.txt file in core currently makes mention of jQuery as being a 3rd party lib. For consistency we should do the same if we're adding a 3rd party JS testing library, too. Just model on what is already there and include that in whatever the next patch is.
I have no technical comments at this time. :-)
#66
Ping. :) Would still love to see this.
#67
With the lack of a working batch API patch, I'd propose that we go back to the simpler approach in #52 - the batch API would be nice to have, but shouldn't be a blocker for the JavaScript testing framework. @boombatower: any thoughts?
#68
@boombatower: If you think we should proceed forward with your batch approach, please give full details of how to demonstrate the failure, and maybe we can get you some help. This is sooooo important. Thanks for all your work on this!
#69
Looking forward to putting some effort into this. Boombatower, please let me know exactly how to make this fail.
#70
Increase the time the js waits before posting results. Batch API will then run faster and continue to re-run the same test over and over since it never received results.
What you need is a flag when you got results or that your waiting..I've tried many variations, short of DB (although I did variable_set|get()). Possibly just make batch API wait longer.
This is extremely close as everything else works great: 1) test detection, 2) re-run, etc.
#71
Great! (subscribing)
#72
subscribe.
#73
subscribing
#74
Unfortunately, if this didn't get in by now it probably won't. Hopefully we can make it for D8.
#75
It would probably be good to start a javascript testing project in contrib (e.g. qunit), before trying to move it into core for Drupal 8. That way, we can have tests for Drupal 7 even if they're not in core, and it would be a great starting point for a JavaScript testing framework for Drupal 8.
Also, I'd like to note that I no longer think that using qunit is the best approach. While qunit is a wonderful automated JavaScript testing framework, my understanding is that it's really designed to test JavaScript independent of web content, not JavaScript that depends on PHP-driven web content (such as collapsible fieldsets, sticky table headers, tabledrag, and nearly every other major JavaScript implementation in Drupal). I think we should work out the details of a testing framework in contrib before coming in with a core patch.
#76
Agreed, that sounds like a good plan. And thanks for chiming in with your thoughts re: QUnit.
#77
Contrib is a great path forward.
We may want to return to the idea of using Selenium. Selenium is not a standalone approach, but it can test using many different browsers, which is critical for our javascript approach. We're far more likely to get real results. And selenium doesn't require anybody to know any javascript, so perhaps we can get more people involved in the test project.
Selenium would be more of a real-life test environment, so might be useful for a broader range of tests than just javascript/ajax related tests. There are places where the Simpletest environment, for all its glory and value, is too handicapped by using a virtual browser.
The upshot is that when we create a contrib project it's worth considering calling it something other than "jstest" or something, because it might have broader applicability.
#78
I really do not see the use of Selenium for Drupal, especially considering the extreme complexity it adds (actually running in all environments) (much less to add support for testing bot - having an X-server and/or windows running for IE and the like).
Drupal runs on top of jQuery and we should assume cross browser support (since they claim it), any cross-browser issues should be rare and/or css/html related. We just need a way to test all the ajax and drupal functionality built on top of jquery.
Having a simple script that runs in browser makes sense and would be much simpler to run on testbot.
I have spent a fair amount of time looking at Selenium and it was quite hard to actually get it going for what we would need for testbot and such (actually gave up). We have taken the DrupalWebTestCase approach for the majority of our functional testing needs...thus we just need to test the javascript components and form alterations.
We need to test one implementation of ajax autofill callback and the rest just need functional tests to ensure they spit out right data...etc.
#79
Selenium *is* a not-so-simple script that runs in the browser. It can also be extended to a far larger testing environment.
But I have *no* experience with it so cannot make the case. It was raised early in this issue (or a related one) and I know it's highly thought of in many circles for doing *one* of the things we need here, which is actual browser experience testing.
As far as javascript unit testing, I think we should use something like the path this was already taking. As far as AJAX forms and the like, we should keep our minds open and discuss the possibilities.
#80
I just wanted to add my 2 cents here.
Looks like John Resig has been working on this issue for a while: http://wiki.github.com/jeresig/testswarm/
a demonstration is here: http://vimeo.com/6281121
Maybe this will be a good solution to testing javascript in a continuous and automated way.
#81
your .02 were added already in #28.
#82
Ah, so it is. I just did a keyword search on this page to see if TestSwarm was mentioned.
#83
We can in fact use an automated Javascript Unit testing framework, which is the title of this issue. The QUnit would do it, and the work done here would actually work.
The problem is that that's not all we need. It's an incredibly small part of what we need. We also need something that tests the interaction of javascript with server-side code. Functional testing of the Drupal experience. How will our confidence increase in AJAX behavior (which is mostly HTML replacement by javascript, and has highly complex FAPI interactions) by just doing Javascript unit testing? There are places where just Javascript unit testing would improve our test situation, but we need actual in-the-browser functional testing of many, many places in Drupal. Every place that uses AJAX. Many more where the interaction between the browser Javascript and the server PHP is significant.
So... QUnit/Testswarm alone will not get us where we want to go. It would not tell us (I don't think) when install.php was broken, or when Poll module was broken. Those involve extensive interaction between the HTML in the browser, the javascript in the browser, and the server-side PHP.
Selenium might, if we could make it do what we want, and find a way to deal with the fact that it's a fundamentally different thing than our current testbot. It may be that we have to use two different approaches to get more coverage. I know that's not optimal, but having some kind of functional coverage of vast swaths of our system is critical.
Are there approaches other than Selenium that might give us true functional testing?
#84
subscribe
#85
I posted an alternative route we could pursue for JavaScript testing at #775050: JavaScript testing framework. I decided to scrap QUnit and write a Drupal-specific framework, but the fundamental concepts are the same. I didn't want to hijack this issue for that purpose, though.
#86
Hope all who are interested in this problem and who are at Drupalcon can come to a BoF today on Javascript functional testing, 4pm, Room 200.
#87
Working on re-applying this, currently manually merging:
- modules/simpletest/simpletest.module
- modules/simpletest/simpletest.pages.inc
- scripts/run-tests.sh
#88
Initial attempt at re-merge.
#89
@cwgordon7, I think you should post your work here. It may be a different approach, but it's still aiming to solve the same problem.
#90
I'm rerolling the patch based on the patch at #62 at its date, rebased with git to the actual HEAD, so now it do not have WSODs but it still needs work because it end up in a loop at js tests on progress bar(batch api implementation bug?).
I'm also getting:
Notice: Array to string conversion in rdf_preprocess_image() (line 720 of /modules/rdf/rdf.module).after running a php test
#91
My current thinking is that a Selenium-based approach will be more robust: #825436: Create selenium-RC PIFR plugin for full functional testing. Your comments and participation are welcome!
#92
Downgrading all D8 criticals to major per http://drupal.org/node/45111
#93
I have done some work for integration of Selenium Webdriver (not Selenium RC) into simpletest. That allows to test javascript and upload of files as webdriver works with browsers in more native way (through browser specific drivers) instead of executing javascripts. The repo is here http://drupal.org/sandbox/ygerasimov/1131210
I have also written blog post http://ygerasimov.com/integrating-selenium-to-drupal-simpletest-framework with video demo http://www.youtube.com/watch?v=4QGY3hxt9Qs
I would very appreciate review and comments about it.
#94
The last submitted patch, 237566-qunit-v7.patch, failed testing.
#95
I fixed up the QUnit module a bit. cwgordon7's GSOC application was also accepted. So good happenings all over the place! Would be nice to get everything working in SimpleTest though, just not sure how, maybe cwgordon7 is onto something.
#96
@Rob Loach, approach from #93 already works with simpletest and can test javascript
#97
...coming from #1221442: Add support for jQuery/JavaScript automated tests.. I honestly don't know what's wrong with me and how this never came up when I searched.
#98
lol.. i think this thread will be so interesting.. (subscribing..)
#99
I have taken over Selenium project and now continue all development there.
#100
how about https://github.com/mishoo/UglifyJS where jquery currently uses.
#101
subscribe
#102
For those not following it, the Selenium project now includes tests for autocomplete.js, collapse.js, machine_name.js, states.js, tabledrag.js, tableheader.js, tableselect.js, textarea.js, vertical_tabs.js.
#103
Awesome! Also note that cwgordon7, jzacsh and I opened up http://drupal.org/project/qunit .
#104
Awesome!
#105
Have any of you had experience with JsTestDriver (http://code.google.com/p/js-test-driver)? I've been using it for a while now and really like it.
I don't have experience setting up testing systems on the scale we're talking about here but I hadn't seen anyone mention it and was curious if it's a viable option.
#106
So it seems that progress is being made on the Selenium and qUnit fronts. I think it may be time to hash out which solution should be branded the "Drupal Way". This is a conversation we need to have in order to establish a standard practice for ensuring that new and current javascript that we include into Drupal 8 is covered by unit tests.
Currently we are looking to refactor javascripts to use the updated syntax and features of jQuery 1.7 so this is heavy on my mind.
I'd like to start adding unit tests to all of our JavaScript so I can be sure that any refactoring doesn't break anything. But I'm not sure which testing framework I should pour my time into. The choice seems to be a toss up. Selenium and qUnit both seem to have momentum behind them.
#107
Knee-jerk question: Which one is used by more other projects, specifically projects we're likely to integrate with? (jQuery, jQuery UI, Symfony2, not sure what else is likely, etc.)
#108
jQuery uses QUnit.
#109
Relevant to this topic: http://www.youtube.com/watch?v=Mbt6h1BFW8g
That video explains what Paul Irish had to do in order to cover Modernizr with unit tests using qUnit.
Also see rfay's post above on his thoughts on Selenium.
Video reference was first found by @jacine.
#110
jQuery itself uses QUnit, and since we require both unit tests (i.e., testing Drupal.checkPlain()) and functional tests (i.e., making sure the taxonomy autocomplete works), it seems to make more sense to use QUnit, which allows us to create both types of tests. I did a lot of work over the summer on the Drupal QUnit project, and ksenzee is continuing that work. Furthermore, using qunit, it's possible to have a fully distributed multiple-browser/multiple-os automated testing system wherein the testbot would just create a page at some location, and then would broadcast that location to many distributed clients which would view the page and execute the tests. The test results and browser/os data could then be reported back to the testbot upon completion, which could update drupal.org with the results.
#111
If jQuery is using qunit, and qunit will do what we want to do, that seems like the obvious choice to me. That makes it easier for people to transfer knowledge back and forth from jQuery in general, which should make life easier for JS developers coming in.
(Note: I say this having never used either library myself, so take with a suitable quantity of sodium chloride.)
#112
Just to chime in again:
QUnit is fantastic for testing javascript code, and would be the obvious choice for that application.
The issue we have to recognize is that nobody has suggested a way that QUnit can do testing of website applications, so we probably need Selenium (or an alternative) as well. We *need* to test things like AJAX Forms, which IMO to be tested properly need the full server/javascript/client environment.
Just a few issues that would really make you scratch your head about how to do it in a pure JS environment:
#309088: In autocomplete textfield, have to hit enter once to accept autocomplete, another time to submit form
#557284: AHAH/AJAX bindings do not work on checkbox or radio in IE6/7/8 (and now IE9)
#1018714: Image Upload Widget Not Working in IE8
#1152848: Can no longer delete images from field in node on production site
#113
If we had selenium testing, we could also try to deprecate some of the browser tests we're doing in PHP in favour of full functional testing. As long as we keep the same level of code coverage it would help us to focus the PHP tests in core on unit and integration tests, whereas we currently have unit, integration and functional tests all bundled together.
#114
Yeah, ideally we use PHPUnit and QUnit for PHP/JS unit testing, respectively, and Selenium for functional testing. Those are each the best tools for the job.
That means learning 3 different testing frameworks for core contributors though, which is quite steep. :\ SimpleTest is nice because it can do both unit and functional tests (on the backend, anyway) without learning extra stuff.
#115
Wait are we talking about supplementing Simpletest with qUnit or are we talking about replacing Simpletest with qUnit? I thought we were talking about supplementing.
I think that qUnit is capable of handling anything javascript related. That's what we need it for. Review this slidedeck for further insights on how to handle testing ajax requests, asynchronous actions, etc.
http://benalman.com/talks/unit-testing-qunit.html#1
#116
Wait are we talking about supplementing Simpletest with qUnit or are we talking about replacing Simpletest with qUnit?
Supplementing only. The idea is to integrate qunit into the existing testing framework.
I'm a bit torn on the Selenium question. My instinct says we should be able to do JS-enabled functional tests with the qunit module and keep people from having to learn a third framework. However, I don't have any such tests written, so I can't prove it. Selenium is certainly the standard for functional testing, and I haven't been involved in any of the work going on there, so I don't know how much my opinion counts for. Just throwing it out there.
#117
It sounds like we just need to go through the effort of writing a few tests so that we can gain some insight on what it takes to write functional tests for Drupal using selenium and qUnit.
Should we focus on the issues that rfay posted above for this purpose?
#118
Selenium can upload files. This is not possible with qUnit or any other javascript testing framework.
#119
Selenium is great for functional test. But we should still have qunit in core. Especially for unit testing our JS. It shouldn't be selenium or qunit it should be qunit for sure. and Selenium if we can :)
#120
But qUnit can mock the contents of an uploaded file. and for some tests this is sufficient.
#121
Seems like a lot of this js related stuff in http://blog.boombatower.com/drupalcon-sf-quality-assurance-thoughts is still relevant or may be of interest.
#122
This makes a lot of sense to me. So far using the Selenium module has been fairly seemless in terms of writing tests after learning SimpleTest. In case anyone hasn't seen an example take a look at the attachment. The whole thing feels like a SimpleTest test with a few extras. This one does drag and drop table re-ordering.
<?php
// Create user.
$account = $this->drupalCreateUser(array('administer taxonomy'));
// Login user.
$this->drupalLogin($account);
// Drag first term down and save.
$this->drupalGet('admin/structure/taxonomy/' . $vocabulary->machine_name);
// Get tid of the first element using hidden form element.
$first_term_tid = $this->driver->getElement('css=tr.draggable.odd .term-id')->getValue();
$first_element = $this->driver->getElement('css=tr.draggable.odd a.tabledrag-handle');
$size = $first_element->getSize();
// Move element down on its size + 5 pixels.
$first_element->dragAndDrop(0, $size['height'] + 5);
$this->verboseScreenshot();
// Wait till drag and drop warning message appear.
$this->driver->waitForElements('css=div.tabledrag-changed-warning div.tabledrag-changed');
// Submit.
$this->driver->getElement('css=#edit-submit')->submit();
// Wait till message appear
$this->driver->waitForElements('css=#messages div.messages .placeholder');
$this->verboseScreenshot();
// Get tid of new first term.
$new_first_term_tid = $this->driver->getElement('css=tr.draggable.odd .term-id')->getValue();
// Ensure that name of first term changed.
$this->assertNotEqual($first_term_tid, $new_first_term_tid, t('Weight of the term changed via drag and drop'));
?>
I haven't seen an example of a qUnit test that does functional testing, but to me it makes sense write functional tests in something that is more similar to the PHP unit tests (PHP and SimpleTest syntax) and to have javascript unit tests in qUnit (see second attachment — or jsTestDriver, which uses qUnit syntax).