Problem/Motivation
There are a few problems with our release/jenkins scripts right now.
1. they are dispersed in 4 different places (debian/rules, release.sh, aegir jenkins scripts, vagrant scripts for aegir)
2. we use sometimes fabric, sometimes puppet, we should choose one or the other (#1436006: merge with "Vagrant Scripts for Aegir" project)
3. the official release process documentation tells us to use release.sh, but there's a changelog task in jenkins that does the same
4. we have multiple branches for 1.x, 2.x, this won't scale to multiple releases and makes it hard to port changes accross versions - we should merge all those branches and the scripts should support all releases with arguments
There are probably other things i'm missing here.
Proposed resolution
We want to refactor all tests to use puppet-aegir and a common codebase.
Remaining tasks
Smoke tests are in place in http://ci.aegirproject.org/view/Puppet/ - they just validate the manifests.
So we have to re-create the tests that were destroyed in the last jenkins upgrade (see #2078531: ci.aegirproject.org loses jobs after every reboot).
We'd need to create a gigantic matrix test, with the following parameters:
* basebox distro (ubuntu 12.04, debian squeeze, debian wheezy)
* git + hostmaster-install vs debian package
* debian: 1.x, 2.x, 3.x (aegir, aegir2, aegir3)
* debian.aegirproject.org: stable/testing/unstable
* git: which ref? tag or branch
we also need:
* a job to automatically build debian packages and upload them to unstable (for 1.x, 2.x, etc)
* a job to do automatic upgrade tests (another matrix?)
those tests can either be triggered or ran daily.
one test that should be triggered at every commit would be to:
* run "update" (git pull and run tests) on a running VM
User interface changes
A much simpler set of tests.
API changes
N/A.
Related Issues
There are too many fields here.
Comments
Comment #1
ergonlogicOver the past couple weeks I've re-factored puppet-drush and puppet-aegir pretty significantly to:
I've also added a
tests/.ci
directory in these puppet modules that contain a Vagrantfile and some scripts. These are set up to run the smoke tests from a "shell provisioning" script. It's a bit of an abuse of Vagrant, but it makes configuring Jenkins jobs absolutely trivial (basically, you just symlink the Vagrantfile into the project root), and pretty much self-contained in the puppet modules themselves.I've taken a similar approach in aegir-up to provision a vm, install aegir using puppet-aegir (either from the .debs and 'manually' from the git repos), and then run our provision tests. I'll probably add comparable tests directly to puppet-aegir to run these tests independently of the extras added by aegir-up.
Admittedly, it takes somewhat longer to run tests this way, ~14 minutes from source vs. ~9 minutes previously (though not really an apples to apples comparison). However, we could cut this down significantly by using a basebox with dependencies (apache/nginx, mysql, php, etc.) already installed.
To get more immediate feedback on changes that don't necessarily affect the installation of Aegir itself, we could cut the test time down even further by using a basebox with Aegir already installed from source, and just update it before running tests.
My reasoning for pursuing this approach was largely to consolidate the various installation and test scripts, as well as to test and use the puppet modules more broadly. I'm setting this issue to 'needs review' in order to get feedback from other maintainers.
Comment #2
anarcat CreditAttribution: anarcat commentedthat's exactly what i was looking for. is this committed and live now? if not it should be.
Comment #3
ergonlogicYes, they're under the 'Puppet' tab.
Comment #3.0
ergonlogiclink to related issues
Comment #4
ergonlogicI believe at least some of the tasks in the summary need to happen before 2.0. Although, the crucial ones might already have individual issues. If so, we should list them here, and downgrade the priority again.
Comment #5
helmo CreditAttribution: helmo commentedIs #1380326: Vagrant-based test harness the same test idea as in #1 or an old attempt?
Comment #6
helmo CreditAttribution: helmo commentedThe installation of Aegir 2 by puppet seems to be working (http://ci.aegirproject.org/view/Puppet/).
Both the latest stable debian package and the sources from git.
What we still need:
* install 1.x and upgrade to 2.x ( both debian and from source)
* install 2.0 and upgrade to latest HEAD ( or stable release)
That could turn out to be quite a matrix.
Variables:
* basebox (Debian 7, Ubuntu last LTS, Ubuntu last stable, ...)
* webserver (apache, nginx)
* database? (mysql, maria)
Comment #6.0
helmo CreditAttribution: helmo commentedissue summary
Comment #7
ergonlogicI'm thinking of doing two matix tests to start, one from the git sources and the other from .debs.
Common variables:
* basebox name
* basebox url
* webserver
git/hostmaster-install variables:
* branches 1.x, 2.x, 3.x
.deb-specific variables:
* package: aegir, aegir2, aegir3
* distribution: stable, testing, unstable
Comment #8
ergonlogicOur first matrix test is mostly working! See: http://ci.aegirproject.org/view/Puppet/job/Matrix.deb/
I had to hardcode the box urls in the Vagrantfile, and the current Wheezy one I have isn't working. I think upgrading Vagrant to 1.2+ will fix that though. We're also only testing Aegir 2 right now, since we're dropping support for Aegir 1, and don't have .debs for Aegir 3 yet.
I'll need to get a comparable set of tests running for git, but I'd like to abstract the Vagrantfile and setup/test scripts more, so they can be shared across all tests. That way, each test will only contain the things (puppet manifest, additional test scripts, etc.) that are really specific to it.
This all needs to be properly documented, which I'll do in a couple readme's. Also, the directory structure should be shuffled around to separate smoke tests from functional tests.
Comment #9
ergonlogicOur test appear to be working quite well, but we still have quite a few that we are no longer running. We should perhaps clean them out, to avoid confusion.
Comment #10
helmo CreditAttribution: helmo at Initfour websolutions commentedYesterday I created http://ci.aegirproject.org/view/Stable builds/job/S_aegir-debian-official-3.x as a copy of http://ci.aegirproject.org/view/Stable%20builds/job/S_aegir-debian-offic...
Both jobs failed on the last test to check that the package file made it into the repository.
I say a problem with that and committed a fix to https://github.com/aegir-project/aegir_jenkins_scripts/commits/master
The root cause was better visible in the reprepro logfile, stored in /var/log/incoming.log
I'v now added
Permit: older_version
to the /srv/reprepro/conf/incoming configfile.The log in now much cleaner ... only:
But still a failure.
Comment #11
helmo CreditAttribution: helmo at Initfour websolutions commentedThe new aegir3-provision_3.0~beta1_all.deb packaged now ended up in the morgue dir.
I've added a note to the release process docs:
And now that I've finally bent by brain around this ... my steps:
reprepro@zeus:~$ reprepro remove unstable aegir3-cluster-slave aegir3 aegir3-provision aegir3-hostmaster
Comment #12
helmo CreditAttribution: helmo at Initfour websolutions commented#2859549: Deprecate or move Jenkins
I added https://github.com/aegir-project/aegir_jenkins_scripts/blob/master/READM... with a link to gitlab CI