Problem/Motivation

There are a few problems with our release/jenkins scripts right now.

1. they are dispersed in 4 different places (debian/rules, release.sh, aegir jenkins scripts, vagrant scripts for aegir)
2. we use sometimes fabric, sometimes puppet, we should choose one or the other (#1436006: merge with "Vagrant Scripts for Aegir" project)
3. the official release process documentation tells us to use release.sh, but there's a changelog task in jenkins that does the same
4. we have multiple branches for 1.x, 2.x, this won't scale to multiple releases and makes it hard to port changes accross versions - we should merge all those branches and the scripts should support all releases with arguments

There are probably other things i'm missing here.

Proposed resolution

We want to refactor all tests to use puppet-aegir and a common codebase.

Remaining tasks

Smoke tests are in place in http://ci.aegirproject.org/view/Puppet/ - they just validate the manifests.

So we have to re-create the tests that were destroyed in the last jenkins upgrade (see #2078531: ci.aegirproject.org loses jobs after every reboot).

We'd need to create a gigantic matrix test, with the following parameters:

* basebox distro (ubuntu 12.04, debian squeeze, debian wheezy)
* git + hostmaster-install vs debian package
* debian: 1.x, 2.x, 3.x (aegir, aegir2, aegir3)
* debian.aegirproject.org: stable/testing/unstable
* git: which ref? tag or branch

we also need:

* a job to automatically build debian packages and upload them to unstable (for 1.x, 2.x, etc)
* a job to do automatic upgrade tests (another matrix?)

those tests can either be triggered or ran daily.

one test that should be triggered at every commit would be to:

* run "update" (git pull and run tests) on a running VM

User interface changes

A much simpler set of tests.

API changes

N/A.

There are too many fields here.

Comments

Status:Active» Needs review

Over the past couple weeks I've re-factored puppet-drush and puppet-aegir pretty significantly to:

  • use parameterized classes, and otherwise modernize the code;
  • remove external dependencies;
  • improve support for Ubuntu & Nginx;
  • add smoke-tests to ensure manifests compile

I've also added a tests/.ci directory in these puppet modules that contain a Vagrantfile and some scripts. These are set up to run the smoke tests from a "shell provisioning" script. It's a bit of an abuse of Vagrant, but it makes configuring Jenkins jobs absolutely trivial (basically, you just symlink the Vagrantfile into the project root), and pretty much self-contained in the puppet modules themselves.

I've taken a similar approach in aegir-up to provision a vm, install aegir using puppet-aegir (either from the .debs and 'manually' from the git repos), and then run our provision tests. I'll probably add comparable tests directly to puppet-aegir to run these tests independently of the extras added by aegir-up.

Admittedly, it takes somewhat longer to run tests this way, ~14 minutes from source vs. ~9 minutes previously (though not really an apples to apples comparison). However, we could cut this down significantly by using a basebox with dependencies (apache/nginx, mysql, php, etc.) already installed.

To get more immediate feedback on changes that don't necessarily affect the installation of Aegir itself, we could cut the test time down even further by using a basebox with Aegir already installed from source, and just update it before running tests.

My reasoning for pursuing this approach was largely to consolidate the various installation and test scripts, as well as to test and use the puppet modules more broadly. I'm setting this issue to 'needs review' in order to get feedback from other maintainers.

that's exactly what i was looking for. is this committed and live now? if not it should be.

Yes, they're under the 'Puppet' tab.

Issue summary:View changes

link to related issues

Priority:Normal» Critical
Status:Needs review» Needs work

I believe at least some of the tasks in the summary need to happen before 2.0. Although, the crucial ones might already have individual issues. If so, we should list them here, and downgrade the priority again.

Is #1380326: Vagrant-based test harness the same test idea as in #1 or an old attempt?

The installation of Aegir 2 by puppet seems to be working (http://ci.aegirproject.org/view/Puppet/).
Both the latest stable debian package and the sources from git.

What we still need:
* install 1.x and upgrade to 2.x ( both debian and from source)
* install 2.0 and upgrade to latest HEAD ( or stable release)

That could turn out to be quite a matrix.

Variables:
* basebox (Debian 7, Ubuntu last LTS, Ubuntu last stable, ...)
* webserver (apache, nginx)
* database? (mysql, maria)

Issue summary:View changes

issue summary

I'm thinking of doing two matix tests to start, one from the git sources and the other from .debs.

Common variables:
* basebox name
* basebox url
* webserver

git/hostmaster-install variables:
* branches 1.x, 2.x, 3.x

.deb-specific variables:
* package: aegir, aegir2, aegir3
* distribution: stable, testing, unstable

Our first matrix test is mostly working! See: http://ci.aegirproject.org/view/Puppet/job/Matrix.deb/

I had to hardcode the box urls in the Vagrantfile, and the current Wheezy one I have isn't working. I think upgrading Vagrant to 1.2+ will fix that though. We're also only testing Aegir 2 right now, since we're dropping support for Aegir 1, and don't have .debs for Aegir 3 yet.

I'll need to get a comparable set of tests running for git, but I'd like to abstract the Vagrantfile and setup/test scripts more, so they can be shared across all tests. That way, each test will only contain the things (puppet manifest, additional test scripts, etc.) that are really specific to it.

This all needs to be properly documented, which I'll do in a couple readme's. Also, the directory structure should be shuffled around to separate smoke tests from functional tests.