If we need full unit test coverage for code, I would assume this includes JavaScript.
I made a patch which has a light-weight unit-testing framework for JS.
Currently it has no UI, but you can run Drupal.test() through firebug.

Also, the checkPlain tests don't pass, and this is for good reason, because checkPlain is broken (yay tests!).

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

dmitrig01’s picture

The fix to the checkPlain bug is http://drupal.org/node/237577, and I have a new patch, which doesn't do stuff in drupal.js but moves tests over to tests.drupal.js.

I will file separate patches for more tests.

quicksketch’s picture

Subscribe (thanks dmitri).

dmitrig01’s picture

I have found out a way to do functional testing. Now we need naming conventions for JS files

catch’s picture

This looks great! Subscribing.

moshe weitzman’s picture

This looks terrific. Should we build upon the testing library of jquery - either testrunner or jqunit? Maybe you did already since they look similar.

Freso’s picture

This is awesome. Thank you, Dmitri! (Subscribing. :))

beeradb’s picture

subscribing - thanks for the great work so far.

dropcube’s picture

subscribing...

Dries’s picture

Is this something that we could invoke from within SimpleTests so that with one click on a button, we can run all tests, including the JS tests.

moshe weitzman’s picture

Jquery has promoted its QUnit unit test framework to be an independant project. They *want* other projects to use this. We should consider it strongly. See http://docs.jquery.com/QUnit

nedjo’s picture

Thanks for taking the lead dmitrig01.

Yes, there are big potential advantages to using the QUnit framework, since it is the standard for jQuery, is under active external development, and has documentation and samples in place.

That said, (a) the QUnit documentation is sketchy so far, (b) Dmitri's version is coded more closely to our existing Drupal js codebase (c) there look to be some methods in this patch not available in QUnit, e.g., Drupal.tests.randomName.

Maybe rework the patch to build off of QUnit while adding missing methods (could also contribute as patches to QUnit).

@Dries, I guess we need to add a JS behaviour that calls JS tests? There are several significant challenges:

* As most (all?) of our JS behaviours depend on specific page content, we have the challenge of rendering that content and adding needed JS and CSS (possibly inline).
* Some JS behaviours will require specific user interaction and data (e.g., the ahah.js behaviors like file uploads).

Do we visit each page with a behaviour in sequence? Do we try to render on a single page all the minimal content needed by all behaviours? I guess the latter is more in line with what we do in other tests. We would require the simple tests to render the HTML needed for a test. Following the upload with AHAH example, this might be something like:


global $conf;
// Cache any existing variable.
$cached = $conf;
// Explicitly allow uploads with stories.
$conf['upload_story'] = TRUE;

// Build and render the upload form.
// Generate a dummy form.
$form = array();
// Upload's form_alter requires a #node property.
$node = new StdObj();
$node->type = 'story';
$form['#node'] = $node;
drupal_alter('form', $form, array(), 'story_node_form');
$form_state = array('submitted' => FALSE);
$form = form_builder('story_node_form', $form, $form_state);
// Rendering the form will ensure we have the appropriate AHAH JS/CSS loaded.
$output = drupal_render($form);

// Restore the previous $conf values.
$conf = $cached;

return $output;

But we're still stuck with the problem of emulating user interaction. Iinitially at least, we might want to limit ourselves to higher level stuff that doesn't involve user actions.

dmitrig01’s picture

Hm, I think it might actually be a good idea to have them each on separate pages, and when defining a JS test in PHP (I was thinking of hook_jstest or something), you could have a callback that would get rendered into an iframe where all the assertions would happen.

RobLoach’s picture

There must be a way we could get this working nicely in a SimpleTest test.

Morbus Iff’s picture

Subscribing.

Status: Needs review » Needs work

The last submitted patch failed testing.

lilou’s picture

Status: Needs work » Needs review
chx’s picture

bdragon’s picture

Subscribing.

jhedstrom’s picture

subscribing

stewsnooze’s picture

subscribing

katbailey’s picture

I made a minor change to dmitri's patch so that Drupal.tests.testAttachBehaviors works with the new Drupal.behaviors.

Status: Needs review » Needs work

The last submitted patch failed testing.

RobLoach’s picture

RobLoach’s picture

cchan’s picture

I'm not sure if this is entirely relevant, but there are automated browser testing frameworks available that emulate user interaction. Selenium (http://seleniumhq.org/) [multiple languages] and Celerity (http://celerity.rubyforge.org/) [Ruby and can run headless] come to mind.

Essentially, you can code, "click on this link and wait for the page to load" or "click on this to see if the drop down box expands". Nifty interactions that are very high level.

It might be useful to run these sorts of tests outside of the simpletest framework - it would probably take a lot of work and code to integrate the two.

I've just started playing around with Celerity, so let me know if any of you guys are interested in pursuing this.

stella’s picture

subscribe

dmitrig01’s picture

@vaeiou selenium and celerity are bad because you have to install them in your browser.

moshe weitzman’s picture

dmitrig01’s picture

i applied for the beta on behalf of drupal

moshe weitzman’s picture

anyone up for this. is a big hole in our test suite. here is an article on qunit just to whet the appetite - http://highoncoding.com/Articles/570_Unit_Testing_JavaScript_Using_JQuer...

Owen Barton’s picture

Subscribe

BTW @dmitrig01 you are incorrect that you need to install Selenium in your browser - you only need to do that if you want to use the GUI test recorder - it is quite capable of running tests standalone (just extract the HTML and scripts anywhere in the webroot) or even driven by, and communicating with a server side script (with tests written in that scripts native language).

rfay’s picture

This is a terribly important subject, as key failures are slipping through the system. A JS introduction to HEAD broke install.php completely the other day.

Let's have a BOF about this at Paris Drupalcon.

mike booth’s picture

Priority: Critical » Normal

Marking as non-critical in accordance with the issue priority guidelines.

rfay’s picture

BOF on this subject today, Wednesday, 2 Sept, 18:00 at Drupalcon.

mike booth’s picture

Priority: Normal » Critical

...aaaand after discussion with webchick we'll mark this as "critical" again, at least until we have a resolution to the "What does Critical really mean" meta-issue. :)

boombatower’s picture

Assigned: dmitrig01 » Unassigned
Status: Needs work » Needs review
FileSize
37.88 KB

cwgordon7 and I work on integrating QUnit with SimpleTest.

boombatower’s picture

FileSize
36.56 KB

Doh

boombatower’s picture

FileSize
36.56 KB

Forgot the actual test suite.

EDIT: having upload issues.

boombatower’s picture

FileSize
43.12 KB

What works:

  • Integration with SimpleTest (Testing) UI
  • Proper re-run tests form
  • Basic javascript unit testing support

What needs to be done in future patches:

  • Proper javascript test selection screen.
  • Related re-run selected functionality.
  • Functional testing framework with iframe, things like get(), post().
  • Creating a separate environment using callbacks for the javascript unit testing framework (proper functional tests)
rfay’s picture

FileSize
3.86 KB
13.42 KB

This is an absolutely fantastic step forward for Drupal. With this patch, we now have the capability of at least doing Javascript unit testing, although we have a bit farther to go to provide functional testing.

As a review, what this patch does *right now*:

  • Adds Javascript unit testing into simpletest
  • Gives a javascript unit testing UI integrated with Simpletest

What it can do with a bit more work (very important work):

  • Functional testing! Test poll module so you can see how it uses AJAX! Test any AJAX situation. Test collapsible fieldsets. YEAH!

Here are a couple of screenshots showing what's been done - A new tab on the tests page for javascript tests, and a list of tests that have been run.

Thanks for the good work, cwgordon7 & boombatower.

sun’s picture

webchick’s picture

Status: Needs review » Needs work

This looks awesome!!! Great, work guys!! :D

A couple of concerns from a quick glance-through:

1. I see verbiage to the effect that code in that new library is BSD. We need to check with Larry about that. I'm pretty sure it works to GPL BSD code, but we might need some fancy stuff.

2. When I click on "Javascript" tab, it initiates an action. This is weird. Every other admin page in Drupal requires an explicit action to trigger it.

Crell’s picture

Status: Needs work » Needs review

cwgordon and webchick asked me about the legal ramifications of including a 3rd party library for JS testing. The best source for that is here, section 2.1:

http://www.softwarefreedom.org/resources/2007/gpl-non-gpl-collaboration....

So if we include QUnit and make no changes to it, then we should include the file verbatim, including all existing copyright notices and information. The licenses on it are GPL-friendly so we're OK.

If we later add a CREDITS.txt file to core, we should include in it references to any of the copyright statements specified in the QUnit file.

sun’s picture

+++ modules/simpletest/simpletest.module	5 Sep 2009 15:10:51 -0000
@@ -86,6 +111,28 @@
+function simpletest_library() {
+  $libraries = array();
+

We can remove that declaration, since we're using string keys.

+++ modules/simpletest/simpletest.pages.inc	5 Sep 2009 15:10:51 -0000
@@ -194,6 +194,73 @@
 /**
+ * User interface for running all JavaScript tests, which is triggered through
+ * the addition of an iframe which runs the JavaScript tests in the browser in
+ * a separate page callback.
+ */
...
+/**
+ * Runs all the JavaScript tests, this is triggered from qunit.php because if
+ * we don't have clean URLs enabled and use a Drupal path, QUnit tries to use
+ * the q GET parameter as a filter for itself.
+ */
...
+/**
+ * JavaScript callback to record JavaScript tests to the database. This is
+ * called once for each JavaScript test.
+ */

Missing PHPDoc summaries here.

+++ modules/simpletest/tests/drupal.test.js	1 Jan 1970 00:00:00 -0000
@@ -0,0 +1,133 @@
+Drupal.tests.testT = {
...
+  test: function() {

Hmm... that additional "test" prefix makes no sense in JS.

Drupal.tests is the namespace already, "t" would be sufficient, because we additionally have the "test" method we try to execute.

+++ modules/simpletest/tests/drupal.test.js	1 Jan 1970 00:00:00 -0000
@@ -0,0 +1,133 @@
+Drupal.tests.testBehaviors = {
...
+  setup: function() {
...
+  test: function() {
...
+  teardown: function() {

I'd recommend to move tearDown right after setUp - additionally, why don't we use the same capitalization as in PHP tests by the way?

+++ modules/simpletest/testrunner.js	1 Jan 1970 00:00:00 -0000
@@ -0,0 +1,803 @@
+/*

Is this entirely copied? If not, then we want to fix the missing second asterisk here.

I'm on crack. Are you, too?

cwgordon7’s picture

  1. Declaration removed.
  2. PHPDoc summaries added.
  3. Prefix removed.
  4. Function moved. We don't use the same capitalization because it is a QUnit function. We can add a wrapper function if we decide it is necessary.
  5. This is entirely copied.

Boombatower will post an updated patch from SVN shortly.

boombatower’s picture

FileSize
54.88 KB

Patch!

Status: Needs review » Needs work

The last submitted patch failed testing.

cwgordon7’s picture

Status: Needs work » Needs review
FileSize
44.48 KB

Sorry about that, test included unrelated portions / old attempts, this one should be good though.

cwgordon7’s picture

FileSize
45.47 KB

Furthermore, here is some refactored JavaScript (that actually works this time), plus increased support for functional testing (I've written functional tests for jquery.once.js and collapse.js, but those belong in separate issues once this is committed -- the collapse.js test actually found a relatively nasty bug, too, which is even more case for having this in core).

I am strongly in favor of committing this sooner rather than later. The sooner this gets in, the more work can be done on JavaScript testing in core, which is critical both for our testing system as well as for our JavaScript code quality. Rfay gave this a solid review, and this patch additionally includes the support for functional testing that he was looking for. I would love to see this get in as soon as possible - I start school very soon, and would love to be able to do work on this while I still have an ample amount of free time.

cwgordon7’s picture

FileSize
45.37 KB

Chx pointed out https.php in IRC and asked form me to make qunit.php more like it. So here is an updated patch. :)

cwgordon7’s picture

FileSize
45.38 KB

And this one fixes having to click the "Run tests" button twice to make the tests run if they've already run. :)

cwgordon7’s picture

FileSize
2.29 KB
5.01 KB

And here are collapse.test.js and jquery.once.test.js, which will go in separate patches later but are here for reference so that we can see how the API works for functional testing. Note that collapse.js.txt is currently broken due to the above mentioned core bug.

boombatower’s picture

Looks like we have some overlapping changes from those I made on plane. I'll try and merge these soon.

boombatower’s picture

FileSize
89.82 KB

Updating the initial list on #40:
What works:

  • Integration with SimpleTest (Testing) UI
  • Proper re-run tests form
  • Basic javascript unit testing support
  • Proper javascript test selection screen.
  • Related re-run selected functionality.

What needs to be done in future patches:

  • Functional testing framework with iframe, things like get(), post().
  • Creating a separate environment using callbacks for the javascript unit testing framework (proper functional tests)

I have cleaned up the backend code significantly. It has been split into three files:

  • simpletest.inc
  • simpletest.js.inc
  • simpletest.php.inc

The overlapping code/base code in simpletest.inc and the specific code for each in their respective files. The JS side now acts EXACTLY like the PHP side from the interface. The only things missing is detection of the JS tests properly...code is setup...but I haven't fleshed out the JS side of things. Right now the three JS tests are hardcoded in detection block.

Also note I have yet to update run-tests.sh as there are a couple of relevant changes. So bot will get a weird result.

More detailed list of changes:

  • Cleaned up REGEX for finding js test files in tests directory.
  • Cleaned up file finding code and separated into function.
  • Moved javascript related code to separate file (probably do same for PHP in followup).
  • Fixed malformated HTML in QUnit test runner.
  • Moved the QUnit test runner to template.
  • Fixed syntax errors in drupal.test.js
  • Cleaned up URL schema.
  • Corrected re-run pointing to conformation form instead of just re-running.
  • Cleaned up no-tests display in table.
  • Abstracted test selection form for use by both JS and PHP.
  • Proper sorting by during cache generation not form generation.
  • Remove code that invoked getInfo() in places where it could just search the cached array.
  • Removed JS redirect since batch API takes care of the logic.
  • Removed extra post variables during JS record.
  • Removed testsuite.css since we don't care how original QUnit results look.
  • Added setting 'activeTest' to alter JS which test to run instead of them all so that batch API actually controls process.
  • General cleanup of a number of bits of code.
  • Cleaned up and corrected logic in the detection of JS tests. Also made it so that is scans all modules not just enabled ones..so that it functions like PHP testing.
webchick’s picture

Subscriiiiiiiiiibe!

Status: Needs review » Needs work

The last submitted patch failed testing.

rfay’s picture

Status: Needs work » Needs review

Resubmit to test bot.

Status: Needs review » Needs work

The last submitted patch failed testing.

kwinters’s picture

I wasn't able to get the tests to run (see below), but code-wise I only saw minor issues. Let me know if I need to do something special to make it work, but ideally it should "just work" on install.

When I run the JS tests, the AJAX call to run the tests doesn't update the counter, and it runs batch?id=2&op=do over and over (I let it run for a long time). This was after updating to most recent HEAD, applying patch, dropping DB and then creating and installing, enabled admin menu and simpletest. "Provide verbose information when running tests" is enabled but I didn't change any other settings. The following was the AJAX response:

{"status":true,"percentage":"0","message":"Processed test 0 of 3 - \x3cem\x3eCheck plain\x3c\/em\x3e.\x3cdiv class=\"simpletest-pass\"\x3eOverall results: 0 passes, 0 fails, and 0 exceptions\x3c\/div\x3e\x3cdiv class=\"item-list\"\x3e\x3cul\x3e\x3cli class=\"first last\"\x3e\x3cdiv class=\"simpletest-pass\"\x3eCheck plain: 0 passes, 0 fails, and 0 exceptions\x3c\/div\x3e\x3c\/li\x3e\n\x3c\/ul\x3e\x3c\/div\x3e\x3ciframe src=\"http:\/\/d7.kernelpanic.local:8888\/modules\/simpletest\/qunit.php\" style=\"display: none;\"\x3e\x3c\/iframe\x3e"}

-files[] = simpletest.install - just making sure this is intentional, not sure what the current state of the registry is regarding install files.

The breadcrumb behavior is a little undesirable: Go to the settings form, then try to go up a level using the breadcrumb. It takes you to the PHP testing start page instead of letting you get to either. This isn't an issue that should hold up a commit though; it's so minor compared to the benefits.

Ensure that at least one tests is selected. singular?

Ensure that the selected tests are valid and that at least one tests is same.

If tests are not cached the goto simpletest/js/collect where wording is strange.

Copyright (c) 2008 John Resig On this line in the testrunner.js file, there are UTF-8 encoding issues. A number of similar issues are further down the file.

Can we just detect and avoid this situation? Infinite loops can be a big pain to debug, and any barrier to test creation is :(

+    // Note: never register a dependency on drupal.js, it is loaded
+    // automatically anyway and will send this into an infinite loop.

test.js has a couple trailing whitespace issues. Also, need $Id$.

simpletest.qunit.tpl.php - I thought that having whitespace before the start tag for html was discouraged, but I could be wrong.

mattyoung’s picture

subscribe

boombatower’s picture

Assigned: Unassigned » boombatower
Status: Needs work » Needs review
FileSize
91.03 KB

Cleaned up some comment, made js test registration actually function (someone may want to look at js since behaviors auto-run so I had to put interesting logic at beginning).

Otherwise all is well, except batch API doesn't display proper numbers when doing js always as the POST requests are asynchronous and it grabs the results before they are fully recorded. I could make code to block, but it seems we would rather run tests faster...perhaps get rid of the results preview when running js tests?

Update run-tests.sh so bot should run.

Probably some other things that could be cleaned...but I think this patch is a big as we should make it (and as small).

As per #60....works fine on fresh install...with no changes other then patch...make sure you clear you caches. It removes and adds the entry in .info...(re-orders to alphabetical)

cwgordon7’s picture

Status: Needs review » Needs work

I unfortunately confirmed the problems pointed out in #60 on a fresh install. The problem is that the iframe that runs the tests is being deleted and reinserted every time the batch API tries to update... so if you have too many tests, or just a slow computer, the iframe is deleted before the JavaScript can finish running the tests and posting the results. There were a few other minor quirks, but this is the only bug that's really preventing this from being committed as-is.

boombatower’s picture

I have attempted this, the batch api change is a peice of cake, but I can't get the proper blocking to work. It is simple logic with $_SESSION, but for some reason the batch API does not see the changes to $_SESSION...if I play with firebug or at random it works...leading me to believe there may be a concurrency issues that happens to screw with session.

I have spent 4-5 hours on this with no dice....taking a break for now.

Crell’s picture

Random addition from legal: The CREDITS.txt file in core currently makes mention of jQuery as being a 3rd party lib. For consistency we should do the same if we're adding a 3rd party JS testing library, too. Just model on what is already there and include that in whatever the next patch is.

I have no technical comments at this time. :-)

webchick’s picture

Ping. :) Would still love to see this.

cwgordon7’s picture

With the lack of a working batch API patch, I'd propose that we go back to the simpler approach in #52 - the batch API would be nice to have, but shouldn't be a blocker for the JavaScript testing framework. @boombatower: any thoughts?

rfay’s picture

@boombatower: If you think we should proceed forward with your batch approach, please give full details of how to demonstrate the failure, and maybe we can get you some help. This is sooooo important. Thanks for all your work on this!

rfay’s picture

Issue tags: +rfaynovember

Looking forward to putting some effort into this. Boombatower, please let me know exactly how to make this fail.

boombatower’s picture

Increase the time the js waits before posting results. Batch API will then run faster and continue to re-run the same test over and over since it never received results.

What you need is a flag when you got results or that your waiting..I've tried many variations, short of DB (although I did variable_set|get()). Possibly just make batch API wait longer.

This is extremely close as everything else works great: 1) test detection, 2) re-run, etc.

casey’s picture

Great! (subscribing)

cweagans’s picture

subscribe.

marvil07’s picture

subscribing

webchick’s picture

Version: 7.x-dev » 8.x-dev
Category: task » feature

Unfortunately, if this didn't get in by now it probably won't. Hopefully we can make it for D8.

cwgordon7’s picture

It would probably be good to start a javascript testing project in contrib (e.g. qunit), before trying to move it into core for Drupal 8. That way, we can have tests for Drupal 7 even if they're not in core, and it would be a great starting point for a JavaScript testing framework for Drupal 8.

Also, I'd like to note that I no longer think that using qunit is the best approach. While qunit is a wonderful automated JavaScript testing framework, my understanding is that it's really designed to test JavaScript independent of web content, not JavaScript that depends on PHP-driven web content (such as collapsible fieldsets, sticky table headers, tabledrag, and nearly every other major JavaScript implementation in Drupal). I think we should work out the details of a testing framework in contrib before coming in with a core patch.

webchick’s picture

Agreed, that sounds like a good plan. And thanks for chiming in with your thoughts re: QUnit.

rfay’s picture

Contrib is a great path forward.

We may want to return to the idea of using Selenium. Selenium is not a standalone approach, but it can test using many different browsers, which is critical for our javascript approach. We're far more likely to get real results. And selenium doesn't require anybody to know any javascript, so perhaps we can get more people involved in the test project.

Selenium would be more of a real-life test environment, so might be useful for a broader range of tests than just javascript/ajax related tests. There are places where the Simpletest environment, for all its glory and value, is too handicapped by using a virtual browser.

The upshot is that when we create a contrib project it's worth considering calling it something other than "jstest" or something, because it might have broader applicability.

boombatower’s picture

I really do not see the use of Selenium for Drupal, especially considering the extreme complexity it adds (actually running in all environments) (much less to add support for testing bot - having an X-server and/or windows running for IE and the like).

Drupal runs on top of jQuery and we should assume cross browser support (since they claim it), any cross-browser issues should be rare and/or css/html related. We just need a way to test all the ajax and drupal functionality built on top of jquery.

Having a simple script that runs in browser makes sense and would be much simpler to run on testbot.

I have spent a fair amount of time looking at Selenium and it was quite hard to actually get it going for what we would need for testbot and such (actually gave up). We have taken the DrupalWebTestCase approach for the majority of our functional testing needs...thus we just need to test the javascript components and form alterations.

We need to test one implementation of ajax autofill callback and the rest just need functional tests to ensure they spit out right data...etc.

rfay’s picture

Selenium *is* a not-so-simple script that runs in the browser. It can also be extended to a far larger testing environment.

But I have *no* experience with it so cannot make the case. It was raised early in this issue (or a related one) and I know it's highly thought of in many circles for doing *one* of the things we need here, which is actual browser experience testing.

As far as javascript unit testing, I think we should use something like the path this was already taking. As far as AJAX forms and the like, we should keep our minds open and discuss the possibilities.

cosmicdreams’s picture

I just wanted to add my 2 cents here.

Looks like John Resig has been working on this issue for a while: http://wiki.github.com/jeresig/testswarm/

a demonstration is here: http://vimeo.com/6281121

Maybe this will be a good solution to testing javascript in a continuous and automated way.

moshe weitzman’s picture

your .02 were added already in #28.

cosmicdreams’s picture

Ah, so it is. I just did a keyword search on this page to see if TestSwarm was mentioned.

rfay’s picture

We can in fact use an automated Javascript Unit testing framework, which is the title of this issue. The QUnit would do it, and the work done here would actually work.

The problem is that that's not all we need. It's an incredibly small part of what we need. We also need something that tests the interaction of javascript with server-side code. Functional testing of the Drupal experience. How will our confidence increase in AJAX behavior (which is mostly HTML replacement by javascript, and has highly complex FAPI interactions) by just doing Javascript unit testing? There are places where just Javascript unit testing would improve our test situation, but we need actual in-the-browser functional testing of many, many places in Drupal. Every place that uses AJAX. Many more where the interaction between the browser Javascript and the server PHP is significant.

So... QUnit/Testswarm alone will not get us where we want to go. It would not tell us (I don't think) when install.php was broken, or when Poll module was broken. Those involve extensive interaction between the HTML in the browser, the javascript in the browser, and the server-side PHP.

Selenium might, if we could make it do what we want, and find a way to deal with the fact that it's a fundamentally different thing than our current testbot. It may be that we have to use two different approaches to get more coverage. I know that's not optimal, but having some kind of functional coverage of vast swaths of our system is critical.

Are there approaches other than Selenium that might give us true functional testing?

chachasikes’s picture

subscribe

cwgordon7’s picture

I posted an alternative route we could pursue for JavaScript testing at #775050: JavaScript testing framework. I decided to scrap QUnit and write a Drupal-specific framework, but the fundamental concepts are the same. I didn't want to hijack this issue for that purpose, though.

rfay’s picture

Hope all who are interested in this problem and who are at Drupalcon can come to a BoF today on Javascript functional testing, 4pm, Room 200.

boombatower’s picture

Working on re-applying this, currently manually merging:

- modules/simpletest/simpletest.module
- modules/simpletest/simpletest.pages.inc
- scripts/run-tests.sh

boombatower’s picture

FileSize
112.33 KB

Initial attempt at re-merge.

cweagans’s picture

@cwgordon7, I think you should post your work here. It may be a different approach, but it's still aiming to solve the same problem.

marvil07’s picture

Status: Needs work » Needs review
FileSize
73.65 KB

I'm rerolling the patch based on the patch at #62 at its date, rebased with git to the actual HEAD, so now it do not have WSODs but it still needs work because it end up in a loop at js tests on progress bar(batch api implementation bug?).

I'm also getting:

Notice: Array to string conversion in rdf_preprocess_image()  (line 720 of /modules/rdf/rdf.module).

after running a php test

rfay’s picture

My current thinking is that a Selenium-based approach will be more robust: #825436: Create selenium-RC PIFR plugin for full functional testing. Your comments and participation are welcome!

catch’s picture

Priority: Critical » Major

Downgrading all D8 criticals to major per http://drupal.org/node/45111

ygerasimov’s picture

I have done some work for integration of Selenium Webdriver (not Selenium RC) into simpletest. That allows to test javascript and upload of files as webdriver works with browsers in more native way (through browser specific drivers) instead of executing javascripts. The repo is here http://drupal.org/sandbox/ygerasimov/1131210

I have also written blog post http://ygerasimov.com/integrating-selenium-to-drupal-simpletest-framework with video demo http://www.youtube.com/watch?v=4QGY3hxt9Qs

I would very appreciate review and comments about it.

Status: Needs review » Needs work

The last submitted patch, 237566-qunit-v7.patch, failed testing.

RobLoach’s picture

Issue tags: +jQuery, +Simpletest, +QUnit

I fixed up the QUnit module a bit. cwgordon7's GSOC application was also accepted. So good happenings all over the place! Would be nice to get everything working in SimpleTest though, just not sure how, maybe cwgordon7 is onto something.

ygerasimov’s picture

@Rob Loach, approach from #93 already works with simpletest and can test javascript

klonos’s picture

...coming from #1221442: Add support for jQuery/JavaScript automated tests.. I honestly don't know what's wrong with me and how this never came up when I searched.

keichee’s picture

lol.. i think this thread will be so interesting.. (subscribing..)

ygerasimov’s picture

I have taken over Selenium project and now continue all development there.

keichee’s picture

how about https://github.com/mishoo/UglifyJS where jquery currently uses.

q0rban’s picture

subscribe

Owen Barton’s picture

For those not following it, the Selenium project now includes tests for autocomplete.js, collapse.js, machine_name.js, states.js, tabledrag.js, tableheader.js, tableselect.js, textarea.js, vertical_tabs.js.

RobLoach’s picture

Awesome! Also note that cwgordon7, jzacsh and I opened up http://drupal.org/project/qunit .

rfay’s picture

Awesome!

sagannotcarl’s picture

Have any of you had experience with JsTestDriver (http://code.google.com/p/js-test-driver)? I've been using it for a while now and really like it.

I don't have experience setting up testing systems on the scale we're talking about here but I hadn't seen anyone mention it and was curious if it's a viable option.

cosmicdreams’s picture

So it seems that progress is being made on the Selenium and qUnit fronts. I think it may be time to hash out which solution should be branded the "Drupal Way". This is a conversation we need to have in order to establish a standard practice for ensuring that new and current javascript that we include into Drupal 8 is covered by unit tests.

Currently we are looking to refactor javascripts to use the updated syntax and features of jQuery 1.7 so this is heavy on my mind.

I'd like to start adding unit tests to all of our JavaScript so I can be sure that any refactoring doesn't break anything. But I'm not sure which testing framework I should pour my time into. The choice seems to be a toss up. Selenium and qUnit both seem to have momentum behind them.

Crell’s picture

Knee-jerk question: Which one is used by more other projects, specifically projects we're likely to integrate with? (jQuery, jQuery UI, Symfony2, not sure what else is likely, etc.)

catch’s picture

jQuery uses QUnit.

cosmicdreams’s picture

Relevant to this topic: http://www.youtube.com/watch?v=Mbt6h1BFW8g

That video explains what Paul Irish had to do in order to cover Modernizr with unit tests using qUnit.

Also see rfay's post above on his thoughts on Selenium.

Video reference was first found by @jacine.

cwgordon7’s picture

jQuery itself uses QUnit, and since we require both unit tests (i.e., testing Drupal.checkPlain()) and functional tests (i.e., making sure the taxonomy autocomplete works), it seems to make more sense to use QUnit, which allows us to create both types of tests. I did a lot of work over the summer on the Drupal QUnit project, and ksenzee is continuing that work. Furthermore, using qunit, it's possible to have a fully distributed multiple-browser/multiple-os automated testing system wherein the testbot would just create a page at some location, and then would broadcast that location to many distributed clients which would view the page and execute the tests. The test results and browser/os data could then be reported back to the testbot upon completion, which could update drupal.org with the results.

Crell’s picture

If jQuery is using qunit, and qunit will do what we want to do, that seems like the obvious choice to me. That makes it easier for people to transfer knowledge back and forth from jQuery in general, which should make life easier for JS developers coming in.

(Note: I say this having never used either library myself, so take with a suitable quantity of sodium chloride.)

rfay’s picture

Just to chime in again:

QUnit is fantastic for testing javascript code, and would be the obvious choice for that application.

The issue we have to recognize is that nobody has suggested a way that QUnit can do testing of website applications, so we probably need Selenium (or an alternative) as well. We *need* to test things like AJAX Forms, which IMO to be tested properly need the full server/javascript/client environment.

Just a few issues that would really make you scratch your head about how to do it in a pure JS environment:

#309088: In autocomplete textfield, have to hit enter once to accept autocomplete, another time to submit form
#557284: AHAH/AJAX bindings do not work on checkbox or radio in IE6/7/8 (and now IE9)
#1018714: Image Upload Widget Not Working in IE8
#1152848: Can no longer delete images from field in node on production site

catch’s picture

If we had selenium testing, we could also try to deprecate some of the browser tests we're doing in PHP in favour of full functional testing. As long as we keep the same level of code coverage it would help us to focus the PHP tests in core on unit and integration tests, whereas we currently have unit, integration and functional tests all bundled together.

webchick’s picture

Yeah, ideally we use PHPUnit and QUnit for PHP/JS unit testing, respectively, and Selenium for functional testing. Those are each the best tools for the job.

That means learning 3 different testing frameworks for core contributors though, which is quite steep. :\ SimpleTest is nice because it can do both unit and functional tests (on the backend, anyway) without learning extra stuff.

cosmicdreams’s picture

Wait are we talking about supplementing Simpletest with qUnit or are we talking about replacing Simpletest with qUnit? I thought we were talking about supplementing.

I think that qUnit is capable of handling anything javascript related. That's what we need it for. Review this slidedeck for further insights on how to handle testing ajax requests, asynchronous actions, etc.

http://benalman.com/talks/unit-testing-qunit.html#1

ksenzee’s picture

Wait are we talking about supplementing Simpletest with qUnit or are we talking about replacing Simpletest with qUnit?

Supplementing only. The idea is to integrate qunit into the existing testing framework.

I'm a bit torn on the Selenium question. My instinct says we should be able to do JS-enabled functional tests with the qunit module and keep people from having to learn a third framework. However, I don't have any such tests written, so I can't prove it. Selenium is certainly the standard for functional testing, and I haven't been involved in any of the work going on there, so I don't know how much my opinion counts for. Just throwing it out there.

cosmicdreams’s picture

It sounds like we just need to go through the effort of writing a few tests so that we can gain some insight on what it takes to write functional tests for Drupal using selenium and qUnit.

Should we focus on the issues that rfay posted above for this purpose?

ygerasimov’s picture

Selenium can upload files. This is not possible with qUnit or any other javascript testing framework.

ericduran’s picture

Selenium is great for functional test. But we should still have qunit in core. Especially for unit testing our JS. It shouldn't be selenium or qunit it should be qunit for sure. and Selenium if we can :)

cosmicdreams’s picture

But qUnit can mock the contents of an uploaded file. and for some tests this is sufficient.

boombatower’s picture

Seems like a lot of this js related stuff in http://blog.boombatower.com/drupalcon-sf-quality-assurance-thoughts is still relevant or may be of interest.

sagannotcarl’s picture

FileSize
1.15 KB
2.03 KB

webchick: Ideally we use PHPUnit and QUnit for PHP/JS unit testing, respectively, and Selenium for functional testing. Those are each the best tools for the job.

This makes a lot of sense to me. So far using the Selenium module has been fairly seemless in terms of writing tests after learning SimpleTest. In case anyone hasn't seen an example take a look at the attachment. The whole thing feels like a SimpleTest test with a few extras. This one does drag and drop table re-ordering.

// Create user.
    $account = $this->drupalCreateUser(array('administer taxonomy'));
    // Login user.
    $this->drupalLogin($account);

    // Drag first term down and save.
    $this->drupalGet('admin/structure/taxonomy/' . $vocabulary->machine_name);

    // Get tid of the first element using hidden form element.
    $first_term_tid = $this->driver->getElement('css=tr.draggable.odd .term-id')->getValue();

    $first_element = $this->driver->getElement('css=tr.draggable.odd a.tabledrag-handle');
    $size = $first_element->getSize();
    // Move element down on its size + 5 pixels.
    $first_element->dragAndDrop(0, $size['height'] + 5);

    $this->verboseScreenshot();

    // Wait till drag and drop warning message appear.
    $this->driver->waitForElements('css=div.tabledrag-changed-warning div.tabledrag-changed');
    // Submit.
    $this->driver->getElement('css=#edit-submit')->submit();
    // Wait till message appear
    $this->driver->waitForElements('css=#messages div.messages .placeholder');

    $this->verboseScreenshot();

    // Get tid of new first term.
    $new_first_term_tid = $this->driver->getElement('css=tr.draggable.odd .term-id')->getValue();

    // Ensure that name of first term changed.
    $this->assertNotEqual($first_term_tid, $new_first_term_tid, t('Weight of the term changed via drag and drop'));

I haven't seen an example of a qUnit test that does functional testing, but to me it makes sense write functional tests in something that is more similar to the PHP unit tests (PHP and SimpleTest syntax) and to have javascript unit tests in qUnit (see second attachment — or jsTestDriver, which uses qUnit syntax).

attiks’s picture

We needed javascript/browser testing for some of our modules, so we created Drupal TestSwarm to run QUnit javascript tests in multiple browsers, it does strict client side testing and reports the findings to the server. All tests are written in javascript as well.

The problem with javascript testing using a headless server is that the number of supported browsers is limited, using the concept of a TestSwarm (like JQuery does) is that you can test your javascript in all browsers (including phones, tablets, ...).

webchick’s picture

Holy CRAP! :D That is SO COOL. Is this in a contributed module somewhere that people can play with it?

attiks’s picture

@webchick: not for the moment, we're still experimenting

New features added to the demo:

  • Test node create as anonymous user, as an example test added to test menu link and update of summary in the tab. The test module disables all submit and validate handlers on the fly.
  • Added an option to run a test manually and see the output of the test, to make debugging easier

Edit: Test link updated

attiks’s picture

Project on d.o. created: http://drupal.org/project/testswarm

The code isn't as clean as is should, will try to do some more cleaning

klonos’s picture

I don't follow all issues that Angie does, but seeing her that excited gets me excited too ;)

...then again this issue goes back to 2008 so the whole excitement makes a *lot* of sense. So many patches/features got reviewed with the fear of breaking things because we simply couldn't have tests for them and they lingered to RTBC for so long because of that fact.

attiks’s picture

@klonos that's part of the reason we created TestSwarm, we experimented with Selenium as well, but the main problem was that it only supports a number of browsers. Selenium has some benefits as well like CI integration, screenshots, test code in php, ...

Next on my own wishlist is adding CI integration to TestSwarm, but I need to figure out how first.

RobLoach’s picture

Nice work... If you push what you changed in the QUnit module to patches in the queue, we could get those in. Ethanw also brought up the possibility of using v8js for it too.

attiks’s picture

@rob that was the intial idea, but the only thing still in common is the qunit.js/css file, the tests in TestSwarm are tight to a page and the necessary javascript is attached to it. The QUnit module (AFAIK) does the testing strictly with javascript code, so it doesn't use the real page. One of our main concerns was to test the javascript in (near to) real life situations, so the tests mimics the real thing as close as possible. Merging both projects will actually mean to abandon one :-)

attiks’s picture

Another proof of concept for 'CI', we cloned testswarm002 (latest stable) to testswarm003 (latest git), primarily to test the fix of #811542: Regression: Required radios throw illegal choice error when none selected

To see the difference:
Latest stable without patch - test will fail
Latest git version - test passes

We also defined a cron job to do regular git pull / drush updb -y / drush cc all to keep it in sync with d.o. git

attiks’s picture

Testswarm now running in auto mode and keeping track of latest git hash of Drupal 7.x. The page refreshes each minute, and runs the tests again whenever a new code got committed.

klonos’s picture

...and the basictests fail in latest firefox nightly :p

attiks’s picture

@klonos: they fail in all browsers, I think it's safe to blame the test, but i need a closer look

attiks’s picture

Test for translations fixed

attiks’s picture

TestSwarm will now test all enabled themes, some tests (drag & drop) will probably fail, since the distance to drag is hard-coded.

rfay’s picture

Issue tags: -rfaynovember

Removing tag. Incredible progress.

klonos’s picture

This is getting better and better!

attiks’s picture

attiks’s picture

#1495672: Automated testing using BrowserStack has landed, test are now run by BrowserStack as well after each commit to Drupal 7

RobLoach’s picture

attiks’s picture

@Rob, added to #1482982: Run QUnit SimpleTests, but casperjs is a wrapper around phantomjs which isn't running as stable as I hoped.

surekha_kolpe’s picture

Status: Needs work » Needs review

#1: javascript_unit_testing.patch queued for re-testing.

moshe weitzman’s picture

Testswarm looks terrific to me as well. Check out its demo site where you can see test results and run them in your own browser (in multiple themes!).

Would be great if someone could work on a patch for bringing Testswarm into Drupal 8.

nod_’s picture

Status: Needs review » Active

nothing to review.

cosmicdreams’s picture

Is there a way forward?

nod_’s picture

Ask the maintainers if they feel they are happy with their code and it's ready to be in core, knowing it will take days to get fixes in afterwards.

cosmicdreams’s picture

nod_: I understand the scope of the problem. I am trying to ascertain the scope of the solution, how I can help, and how to start.

Many solutions have been discussed in this issue. Despite the quality suggestions brought forth here, I still don't know the next step for resolving this issue.

attiks’s picture

As the maintainer of testswarm I would love to see this integrated, but with the current testing infrastructure it's going to be very hard to do. Once the new testing infrastructure is in use it might be possible, but even then it will still be a lot of work.

In the meantime you can always setup you're own testing infrastructure using testswarm. I'm also looking for an easy way to expose my own servers to other Drupal contributors, but haven't found an easy, solid way yet.

Wim Leers’s picture

Version: 8.x-dev » 9.x-dev

Unfortunately.

RobLoach’s picture

As a side note, I've added automated testing to jQuery Once with npm, grunt and QUnit.

cosmicdreams’s picture

Yes, this space (javascript unit testing) is advancing rapidly. Perhaps by D9 a common approach will have emerged.

nicu’s picture

Adding tag

catch’s picture

Version: 9.x-dev » 8.x-dev
Assigned: boombatower » Unassigned
Category: Feature request » Task
Issue summary: View changes

There was a good discussion/session on this at DrupalCon, but that never made it back here.

What I'd personally like us to do, I'm using Behat as an example because I've personally used it on client projects (and because there's several people around core who've also done that), however the same would be possible with a different tool.

* Slowly build up a context class and functional test coverage for JS in core, using Behat. Behat means we get Mink, which can run phantomjs or a 'real' browser via Selenium webdriver.

* Integrate behat with qa.drupal.org

* Apart from absolute basics like logging in, posting content etc., focus on testing core's js so there's usable test coverage whereas we currently have none in core.

* For functional tests where it makes sense (i.e. something like views UI or drag and drop especially), retire SimpleTest tests where thee coverage is 100% replicated with Behat.

* Potentially start using Behat for other functional tests where the browser is a requirement.

Then we're left with two things which aren't covered by PHPUnit or Behat - PHP-only simpletests (DUTB or webtests with no browser usage), and actual unit testing of JavaScript.

I don't see any reason not to add js testing to 8.x, whether that's before or after release, semantic versioning or not - it's entirely a developer tool and has no runtime code implications, so moving the issue back.

clemens.tolboom’s picture

I just learned https://github.com/dreditor/dreditor is using npm + Grunt running QUnit + phantomjs for it's tests.

According to #83 by @rfay we need more then just QUnit.

I'd love to learn more JS testing strategies but the issue summary is not helping or in line with the latest comments. That could have some love first I guess.

attiks’s picture

#156 QUnit alone is indeed not enough, but combined with testswarm it's able to test the backend as well. If we want to test the install as well, we need something like behat.

I used behat the last couple of months and it's probably easier for non-js developers to use, we automated it using jenkins but a bash script could do the same.

#155 I can not agree more, we need to add a solid js testing solution to D8

pbuyle’s picture

Note: I'm coming from http://read.theodoreb.net/2014/critical-gaps-in-drupal-js.html

Unit testing, integration testing and validation testing are three different beasts, looking for a single solution for all three of them is doomed to failure.

QUnit is a unit testing framework for JavaScript, like PHPUnit is for PHP. So if what is needed are integration and validation tests, something more than QUnit is indeed needed. IMHO, for unit testing in JS, neither and a browser or an HTML page is required, but you may need to mock some DOM objects (ie. you don't run your tests over an actual site with real pages). For instance, a properly designed Drupal behavior should be testable by calling it over a set of fixture DOM elements and asserting the wanted changes of these elements or their children.

Being a BDD framework, Behat is a more a validation testing framework. It tests an complete web application as a blackbox, not its individual components. So it could be used to validate that the JavaScript on a standard Drupal install works as expected. As a bonus, Behat is also usable to test behaviors defined by both PHP and JavaScript at the same time. An example of validation testing would be testing that a specific Views admin scenario create the expected view.

That leave us with integration testing, which is hard to implement over code that is not designed as loosely coupled components. Unit testing has the same kind of issue, but while it is still possible to unit test individual functions and classes by mocking their (hidden) dependencies, integration testing requires a known logic to assemble components together in order to be able to tests the various integration scenarios.

My understanding is that event in D8, Drupal's JavaScript code is not designed as components (even tightly coupled), making integration testing nearly impossible to implement. So focus on both unit testing (which the original issue is all about) and validation testing (which has been pointed as needed too) will be more rewarding. Both will encourage more discipline and better quality in the JavaScript code, which should eventually make it possible to implement integration testing.

catch’s picture

Discussed this with Jesse Beach, nod_ and Alex Pott yesterday at Dev Days and we strongly feel we should use Behat to get functional JavaScript test coverage in core. It gives us access to real browsers and phantomjs via mink, the tests are nicely self documenting with gherkin, lots of core developers are already familiar with it via client projects, it's used on Drupal.org etc. etc.

This wouldn't be a replacement for SimpleTest, but likely all the SimpleTests using the browser could be moved over.

There will be things we can't do with Behat, but we can tackle those separately.

This could use a new issue to sort out what the implementation might look like.

cosmicdreams’s picture

Very excited to hear that. Love Behat. Love it's ability to express what a test is testing in real human readable words / paragraphs.

What's the next step?

nod_’s picture

Opened #2232271: [Meta] Use Behat for validation testing, let's close this one and start fresh.

@mongolito404: can you post your comment in the new issue ?

catch’s picture

Status: Active » Closed (duplicate)