One of the main challenges when working on Panopoly (even with something as simple as a contrib module upgrade) is making sure that by fixing one thing, you didn't break something else. Manual testing is never comprehensive and can only catch so much.

The eventual goal, is to have a full suite of Behat tests that are automatically run on a new install of Panopoly every time something is committed. I think having this "safety net" for making changes to Panopoly will help to accelerate its development in the future.

This is a meta issue to get us to that goal!

Todo

Comments

mrfelton’s picture

What would envisage that this does, other than build the platform? We use Jenkins quite extensively with our own distributions which does a few things, essentially:

1) Builds the platform
2) Installs a site
3) Runs a set of Selenium tests

dsnopek’s picture

Yeah, it's going to do pretty much the same things but instead of Selenium, it will be some Simpletest and Behat tests (which may at some point use Selenium underneath). I'm going to try and focus on Behat (via the Drupal extension) when possible, so that the tests are human readable.

andrew_mallis’s picture

Apparently, the Canadian government team employing panopoly has already cooked up a lot of this stuff. Perhaps they'd be interested in sharing or imparting wisdom.

@populist?

dsnopek’s picture

@Andrew_Mallis: That is a great idea!

@populist mentioned the Canadian Government project at BADCamp, but I still haven't a chance to look into it. I just Google'd up a reference to it (Web Experience Toolkit) but I'm not sure still what exactly it is or where it touches Drupal/Panopoly:

http://wet-boew.github.io/wet-boew/index-en.html

But I'll dig in, when I find a chance. :-)

lsolesen’s picture

@dsnopek Did you create any Behat tests yet? Or figured out how to organize those? I could contribute some tests, if you'd like?

dsnopek’s picture

@isolesen: I haven't created any "real" Behat tests yet, I've been mostly experimenting with connecting all the peices together. However, I'm giving a presentation on Behat at my local Drupal meetup on Thusday (http://www.meetup.com/Drupal414/events/146920562/) and I'm thinking about trying to organize a Panopoly Behat test sprint with the same group in early December, but I have to run the idea by them first and see if there is enough interest. :-)

It would amazing if you would contribute some tests! Sometime next week, I'll start committing the infrastructure to run them to Panopoly and we can start creating some real tests. :-)

dsnopek’s picture

Just a note for myself or anyone who was curious about the Canadian government distribution...

I finally found the Panopoly-based distribution created by the Canadian government entirely by accident (they were posting about an issue with Drupal.org in the issue queue) and I wanted to post the link so I wouldn't lose it:

https://drupal.org/project/wetkit

lsolesen’s picture

Some information about how to setup some CI using github, travis and behat can be found here:

http://affinitybridge.com/blog/testing-drupal-distributions-using-behat-... with code examples here https://github.com/sprice/classic

And more from the same company http://affinitybridge.com/blog/we-talked-about-behaviour-testing-and-con...

dsnopek’s picture

The first step to this is setting up the boilerplate for Behat tests in Panopoly: #2145447: Add initial Behat tests to Panopoly

@lsolesen: Thanks for the links! I've been working on an approach for this in an other project, but I take a look at those articles before I commit anything to see if there's anything worth borrowing from them. :-)

To everyone: Sorry, I haven't committed this yet! I meant to last week but things got busy. :-) I'll try and do it this week for sure.

dsnopek’s picture

Title: Setup continuous integration server for Panopoly » [meta] Implement continuous integration for Panopoly
Issue summary: View changes
Related issues: +#2145447: Add initial Behat tests to Panopoly

I made this into "meta issue" - we can create sub-issues for the individual tasks as they come up.

dsnopek’s picture

Issue summary: View changes

The next step is compiling a list of Panopoly's features that we want to make sure we don't break! I've added a new issue which links to Google Doc we can collaborate on: #2146747: Make a list of "core features" to Panopoly that we don't want to break

dsnopek’s picture

dsnopek’s picture

Issue summary: View changes

(Removed note about myself offering servers for this since we might be able to do it with Pantheon and servers from SystemSeed that already have Jenkins/Selenium going - we'll work that out in this issue: #2146755: Setup Jenkins to build Panopoly, install and run the tests for each commit)

dsnopek’s picture

Issue summary: View changes

Adding issue for the Behat tests of Panopoly widgets.

dsnopek’s picture

Issue summary: View changes
dsnopek’s picture

Issue summary: View changes
dsnopek’s picture

Issue summary: View changes

Added the travis-ci issue.

dsnopek’s picture

Status: Active » Fixed

Implemented! I'll let the remaining issues speak for themselves.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.