Create selenium-RC PIFR plugin for full functional testing [#825436]

After lots of discussion on Javascript and functional-page testing, it seems like Selenium would be a possible route, and it would fit right into the PIFR framework.

First, a little background on how Selenium works in the PHP world.

Selenium is a Java-based full-functional-browser testing framework.
Its tests can be written in a number of languages, one of which is PHP.
PHP-implemented Selenium tests are executed by the phpunittest-runner framework. If running on the command-line, you'd do something like "phpunit MyTest" to run the tests in the MyTest class, or "phpunit /tests" to run all the tests in the "tests" directory. It's very similar to our own, familiar run-tests.sh. (details on phpunit and selenium).
When a Selenium test is run in phpunit, it tells how things are done in the setUp(). $this->selenium = new Testing_Selenium("*iexplore", "http://www.google.com", '192.168.5.109'); would inform the Selenium library to use IE, go to Google.com, and use the Selenium RC server on 192.168.5.109. (which could be a Windows machine).
PHPUnit returns success and failure details, as other test-runners, including our own, do.
The Selenium project also provides Selenium IDE, a slick little Firefox Add-on that helps automate the generation of (very simple) tests. It's good for the basis of tests, and makes creating tests a bit more accessible to people who wouldn't use it otherwise. But it's really just a starting point.

A new "Selenium" plugin type (or perhaps it should be a phpunit plugin type) could basically imitate what we're doing with Simpletest or Selenium.

It would have its own test type (.phpunit-test?)
When patches are submitted, they get run against the tests that are available for this type, just as we currently run .test files against the simpletest plugin.
An initial implementation would just be a headless Linux box running X that could launch firefox and run it through the tests.
It can return details with a pass/fail just as we currently do.
A Drupal install could be created for each test using the existing code for it to run against.

Comment	File	Size	Author
#11	selenium.patch	7.95 KB	Anonymous (not verified)

Comments

Comment #1

damien tournoud commented 12 June 2010 at 00:42

Subscribe. Setting up a Selenium instance with a headless Firefox (or Chromium, isn't webkit generally more picky about HTML and Javascript correctness?) is not that much a hassle.

The trickiest part, I believe would be to test that PIFR 2.x actually support multiple testing environments. We never actually made that work until now.

Comment #2

marvil07 commented 13 June 2010 at 06:45

subscribing

Comment #3

boombatower commented 15 June 2010 at 03:46

I see the use of running javascript testing inside headless firefox such as crowbar and/or webkit, but I still don't see the use in the extra complication of selenium.

Comment #4

rfay

he,him

English

Palisade, CO, USA

commented 15 June 2010 at 04:10

Selenium is a robust way of testing browser/server interaction (like AJAX forms). I've done some work with trying to make jQuery run AJAX forms in a test environment (like the two qunit-like scenarios we started with) and it's pretty questionable. Selenium, on the other hand, is designed for this kind of interaction. It would also make Selenium-IDE (which you've worked with) accessible for use with our test workflow.

I'm not sure what you mean about complexity. It seems to me like setting up a few tests and running phpunit and having it do firefox exercises in a headless linux machine is pretty easy.

Comment #5

boombatower commented 15 June 2010 at 20:25

Setting up the actual selenium RC plugins and/or grid is no simple matter. I'm sure someone with more experience might be able to do it easier, but I had minimal success after half day of messing with it. I could get the RC setup, but it never worked as it was described...it left all kinds of browser instances open and didn't like the actually be triggered remotely, only worked if I ran command on that machine manually.

Comment #6

rfay

he,him

English

Palisade, CO, USA

commented 15 June 2010 at 20:34

I spend a bit of time on it and got it going, and I was able to launch remotely as well. I have to admit it took me a fair bit of time to grok it.

And I also have to admit that I don't have any concept of whether it would be reliable, because I don't have enough experience. However, it's basically the leader in the field. And I *think* it would fit right into our model.

Comment #7

Anonymous (not verified) commented 7 July 2010 at 05:40

subscribe. very cool. we just had a commit that broke head that we can't test without some js testing.

Comment #8

Anonymous (not verified) commented 7 July 2010 at 08:22

ok, got hacky integration working, will post code shortly.

i asked webchick about using code licensed "Apache License, Version 2.0", seems that's a no go, so we'd need to ask for dual-licensing from upstream.

anyone know about the Archive_Tar code in modules/system/system.tar.inc ? its BSD, but we're distributing it...

Comment #9

seutje commented 7 July 2010 at 08:32

sub

Comment #10

Anonymous (not verified) commented 7 July 2010 at 11:58

Project:	Project Issue File Review	» Drupal core
Version:	6.x-2.x-dev	» 7.x-dev
Component:	Code	» simpletest.module
Assigned:	Unassigned	»
Status:	Active	» Needs review

ok, hacky hacky patch attached.

to play along at home, you'll need:

a) the selenium php client driver's 'Testing' folder in your path
b) set the variables in FieldUiJavascriptTest to point at your server and child drupal
c) a working selenium RC install

the HMAC cookie stuff could do with some polish, but i think the idea is sound, once we actually invalidate the HMAC based on a timestamp range...

apart from code, we need to see if we can get the driver licensed as GPL or BSD.

Comment #11

Anonymous (not verified) commented 7 July 2010 at 12:56

Status	File	Size
new	selenium.patch	7.95 KB

with patch this time.

Comment #12

7 July 2010 at 13:10

Status:

Needs review

» Needs work

The last submitted patch, selenium.patch, failed testing.

Comment #13

Anonymous (not verified) commented 8 July 2010 at 04:47

talked to rfay in #drupal about this, he asked for a summary of what i was thinking, so here goes:

1. get support for running tests against selenium in D7 core, even if don't have the selenium tests in core. right now that mostly means allowing the db prefix and etc to be changed based on a cookie as well as (or instead of) a UA header. this is a small change, so i'm hopeful i can get that in.

2. integrate selenium tests with simpletest as much as possible. the crappy proof-of-concept code at #11 shows that this is pretty easy. i could be wrong, but this seems like the way to a) allow for minimal changes to the testbot and b) allow for maximum simpletest skill transfer

3. make it easy for devs with a selenium RC install to write tests. i'm thinking it should be as easy as installing a module and setting some config for the selenium host and the dev site host and that's it. this will require packaging some code that can talk to the selenium RC server, either by reusing existing code or implementing it from scratch.

4. see if we can get some best practice going during the D8 cycle, hopefully leading to shipping D8 with js testing in core

Comment #14

kwinters commented 9 July 2010 at 13:21

I have some concerns regarding the ease of use for test developers, which may just be from my lack of exposure to Selenium.

Basically, no matter how great the testing framework is, if it's hard to run the tests in your development environment then it will have poor adoption. That would be very bad for javascript unit testing, and at that point it might only make sense to use Selenium for what other frameworks can't do and we'd still need #237566: Automated JavaScript unit testing framework, too.

I write code on a Mac, and test it locally using MAMP (fairly common practice). What steps are going to be needed to run Selenium tests in this environment? WAMP may be a bigger issue.

What about people who test their code on a dedicated development server? Or a cheap shared server that doesn't let you install things?

"Use the test bot" isn't a full solution. Uploading to the test bot without being fairly confident it will work is a painful process.

Comment #15

rfay

he,him

English

Palisade, CO, USA

commented 9 July 2010 at 13:30

@kwinters, I think all of your concerns are valid. The additional one: Selenium-php (or Selenese PHP) is different from Simpletest even though it looks the same, so there's a whole additional knowledgeset. @JustinRandell hopes to overcome this by wrapping up many similar functions, but that doesn't completely make the problem go away.

We've looked at a number of different approaches to this problem, though, and in my mind this is the best of the crowd. We need in-browser testing. I don't think we're going to get it without some configuration and added complexity.

I should mention that many, many developers now do just use the test bot instead of testing locally, because the bot is so much faster. But I, like you, am uncomfortable with that workflow, and it's ugly in the issue queue.

-Randy

Comment #16

kwinters commented 9 July 2010 at 13:42

I'm not as concerned about coding differences, since that can be mitigated and the people writing tests have control over it. Also, the people affected are mostly just test writers, not as much patch contributors who are getting tested. Getting additional software installed can easily be a pain or effectively impossible.

What are your thoughts re: using Selenium for things that only it can do, and the approach in the other issue for more basic JS unit testing? It could lessen the impact of install issues at the cost of potentially more test system code.

Comment #17

Anonymous (not verified) commented 9 July 2010 at 13:43

@kwinters - needing a selenium RC setup is a big barrier, but this is additional to the tests that use DrupalWebTestCase (at least for now). devs that don't want to set this up and write these tests don't have to change a thing. if even 5% of devs write tests that cover in-browser failures via selenium, we've made progress. we've had a few commits recently that just broke HEAD, and we were none the wiser because only a real browser based test would find it.

at least as i see it, for D7 this will be contrib only. hopefully we can get enough experience and success with selenium to push for the tests to live in core for D8, if not, we'll have learned through experience a lot about what a real solution will look like.

Comment #18

rfay

he,him

English

Palisade, CO, USA

commented 9 July 2010 at 13:54

@kwinters, I am expecting that initially the Selenium approach would be used only where needed, although because of its increased capability it's possible that it could gain favor. As far as the other approaches to JS testing (mostly involving running JS-only tasks in an iframe in the tester's browser), I think they have near-fatal flaws. I spent a lot of time exploring them, and the running-in-an-iframe technique is ugly and fragile. Running in the tester's browser means that we have no way to automate testing, and no control of the browser that's doing it. (Not to say that it wouldn't be an incremental step forward, it's just that a Selenium approach gives us real, automatable functional tests.)

Comment #19

kwinters commented 9 July 2010 at 13:58

I'm all for an experimental "let's see how far we can reasonably take it" approach. If it works then great, I mostly just wanted to make sure someone was thinking about the barrier to entry.

Comment #20

boombatower commented 9 July 2010 at 21:32

As I discussed in my blog post I don't think we need a javascript testing framework anywhere near as complex as selenium and I have found the way in which it runs tests to be overly complex. If we go with the other approach using something like qunit it becomes no harder then running current simpletests for developers and pifr simply needs a webkit or crowbar (geiko engine) to run the javascript.

Comment #21

Anonymous (not verified) commented 10 July 2010 at 01:24

i'm looking into qunit now. i'm starting from the point of view "can it catch bugs like those introduced by recent commits which broke HEAD". it doesn't matter how simple it is if it can't do that.

for example:

whatever we use has to catch this sort of thing. these bugs are what got me looking into this.

Comment #22

rfay

he,him

English

Palisade, CO, USA

commented 10 July 2010 at 02:37

qunit is a just a test runner, and it works pretty well. The real issue is whether we can make a test. If we can, we can do lots of things. I'm all in favor of progress on all of this any way we can get it. However, the approach in #775050: JavaScript testing framework #10-15 was possible. It seemed fragile, but I did testing in an iframe to work. Wrapping the test runner around that is, I think, a solvable problem. Doing it in an automated testing environment seems like something else entirely.