Closed (fixed)
Project:
Panopoly
Version:
7.x-1.0-rc5
Component:
Code
Priority:
Normal
Category:
Task
Assigned:
Reporter:
Created:
5 Nov 2013 at 14:44 UTC
Updated:
8 Feb 2014 at 16:50 UTC
Jump to comment: Most recent
One of the main challenges when working on Panopoly (even with something as simple as a contrib module upgrade) is making sure that by fixing one thing, you didn't break something else. Manual testing is never comprehensive and can only catch so much.
The eventual goal, is to have a full suite of Behat tests that are automatically run on a new install of Panopoly every time something is committed. I think having this "safety net" for making changes to Panopoly will help to accelerate its development in the future.
This is a meta issue to get us to that goal!
Comments
Comment #1
mrfelton commentedWhat would envisage that this does, other than build the platform? We use Jenkins quite extensively with our own distributions which does a few things, essentially:
1) Builds the platform
2) Installs a site
3) Runs a set of Selenium tests
Comment #2
dsnopekYeah, it's going to do pretty much the same things but instead of Selenium, it will be some Simpletest and Behat tests (which may at some point use Selenium underneath). I'm going to try and focus on Behat (via the Drupal extension) when possible, so that the tests are human readable.
Comment #3
andrew_mallis commentedApparently, the Canadian government team employing panopoly has already cooked up a lot of this stuff. Perhaps they'd be interested in sharing or imparting wisdom.
@populist?
Comment #4
dsnopek@Andrew_Mallis: That is a great idea!
@populist mentioned the Canadian Government project at BADCamp, but I still haven't a chance to look into it. I just Google'd up a reference to it (Web Experience Toolkit) but I'm not sure still what exactly it is or where it touches Drupal/Panopoly:
http://wet-boew.github.io/wet-boew/index-en.html
But I'll dig in, when I find a chance. :-)
Comment #5
lsolesen commented@dsnopek Did you create any Behat tests yet? Or figured out how to organize those? I could contribute some tests, if you'd like?
Comment #6
dsnopek@isolesen: I haven't created any "real" Behat tests yet, I've been mostly experimenting with connecting all the peices together. However, I'm giving a presentation on Behat at my local Drupal meetup on Thusday (http://www.meetup.com/Drupal414/events/146920562/) and I'm thinking about trying to organize a Panopoly Behat test sprint with the same group in early December, but I have to run the idea by them first and see if there is enough interest. :-)
It would amazing if you would contribute some tests! Sometime next week, I'll start committing the infrastructure to run them to Panopoly and we can start creating some real tests. :-)
Comment #7
dsnopekJust a note for myself or anyone who was curious about the Canadian government distribution...
I finally found the Panopoly-based distribution created by the Canadian government entirely by accident (they were posting about an issue with Drupal.org in the issue queue) and I wanted to post the link so I wouldn't lose it:
https://drupal.org/project/wetkit
Comment #8
lsolesen commentedSome information about how to setup some CI using github, travis and behat can be found here:
http://affinitybridge.com/blog/testing-drupal-distributions-using-behat-... with code examples here https://github.com/sprice/classic
And more from the same company http://affinitybridge.com/blog/we-talked-about-behaviour-testing-and-con...
Comment #9
dsnopekThe first step to this is setting up the boilerplate for Behat tests in Panopoly: #2145447: Add initial Behat tests to Panopoly
@lsolesen: Thanks for the links! I've been working on an approach for this in an other project, but I take a look at those articles before I commit anything to see if there's anything worth borrowing from them. :-)
To everyone: Sorry, I haven't committed this yet! I meant to last week but things got busy. :-) I'll try and do it this week for sure.
Comment #10
dsnopekI made this into "meta issue" - we can create sub-issues for the individual tasks as they come up.
Comment #11
dsnopekThe next step is compiling a list of Panopoly's features that we want to make sure we don't break! I've added a new issue which links to Google Doc we can collaborate on: #2146747: Make a list of "core features" to Panopoly that we don't want to break
Comment #12
dsnopekAdded a seperate issue for the Jenkins part of this: #2146755: Setup Jenkins to build Panopoly, install and run the tests for each commit
Comment #13
dsnopek(Removed note about myself offering servers for this since we might be able to do it with Pantheon and servers from SystemSeed that already have Jenkins/Selenium going - we'll work that out in this issue: #2146755: Setup Jenkins to build Panopoly, install and run the tests for each commit)
Comment #14
dsnopekAdding issue for the Behat tests of Panopoly widgets.
Comment #15
dsnopekComment #16
dsnopekComment #17
dsnopekAdded the travis-ci issue.
Comment #18
dsnopekImplemented! I'll let the remaining issues speak for themselves.