debian package testing on jenkins

Step #1 done. 5 to go:

2. test installing the package
3. remove the package
4. purge the package
5. install a previous version (last release)
6. install the built package

I have really split this up in two different job chains, to test both the latest release and the "current branch":

A) the "official" chain (or aegir-debian-official in jenkins) builds the package from the current "debian" branch, which is the latest "debian release", e.g. 1.1-4 right now (this is in the Debian rules file, btw: ./debian/rules jenkins-build-official)
B) the "nightly build" chain (or aegir-debian-auto in jenkins) builds the package from the current "debian" branch plus any changes on the 1.x branch (also in the makefile, ./debian/rules jenkins-build-auto)

From there on, steps 2 to 6 are exactly the same. The trick is that those should take arguments, that would be the package to test. In fact, I think we should probably fold jobs in the following way:

1. build package (either auto or official)
2. install, remove, purge package
3. install previous package
4. install built package

In fact, looking at jenkins "tasks" system, I wonder if we shouldn't use that instead of the "build steps". Tasks seem to be the same thing as "build steps" except that they have names and can be reused in other contexts. So for example, this means we would have a job named "official" that would have the following tasks:

1. build
2. install
3. remove
4. purge

Each being a (fairly simple) shell script, for example:

1. ./debian/rules jenkins-build-official
2. dpkg -i $package
3. dpkg --remove $package
4. dpkg --purge $package

Now the trick is that those are reusable, so in effect, the total job would be:

1. build
2. install
3. remove
4. purge
5. (other job's) install
6. install

This needs to be experimented with, but I think that's the gist of it. There are still a few things to decide here before I go ahead with all this:

1. do we install/remove/update/whatnot on the jenkins server? (be scared)
2. do we install using dpkg -i or do we upload to debian.koumbit.net's unstable repository and use apt-get? (the latter has the disadvantage of having to wait a undetermined delay for repo to catchup, but has the upside of broadening the test capabilities)
3. do we use "tasks" or "build steps"? (wtf?)

Also note that if we use apt-get, we have the possibility of using fabric again - by screwing around with the sources.list, we could install the "stable" repository and then upgrade to "unstable" (or whatever).

We have the following technical challenges:

1. how do we propagate packages between tasks/jobs? (may be simply to "upload package to debian.koumbit.net" after the build and then "install package from repository X" within fabric)
2. how do we preseed the stupid mysql root password?

I would also like the current tests performed after install/upgrade to be done within the Debian package, but I think that's another topic - we should have those as tests within the provision repo anyways...

There are fairly good points made in a dupe of this issue here: #1128614: Add .deb-based build to Jenkins scripts, most of which:

1. maybe we don't want to test every commit as a debian package?
2. preseeding the root password doesn't work (as mentionned above)

Ah, and I tested out the batch stuff - basically, *nothing* happens if you only specify a batch task and no "build step", so in effect, those are really "manual tasks" that can be triggered after the build, but are not *part* of the build.

Rather odd, if you ask me, but it seems like it's the way this works. Jobs it will be.

(A potential "task" could be to upload a build package, for example.)

A few good references, this is what we want to do basically:

https://wiki.jenkins-ci.org/display/JENKINS/Splitting+a+big+job+into+sma...

for this we need fingerprinting of the artifacts:

https://wiki.jenkins-ci.org/display/JENKINS/Fingerprint

... so we need to build in . instead of .. so that the artifact stuff can pickup the packages...

Those could upload our packages... maybe:

https://wiki.jenkins-ci.org/display/JENKINS/ArtifactDeployer+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin

This is how we could make a job made out of other jobs through the commandline, and that could parse the output of previous jobs:

https://wiki.jenkins-ci.org/display/JENKINS/Writing+CLI+commands
https://cloudbees.zendesk.com/entries/477213-doing-choreography-workflow...

... not sure we want that...

Almost off topic here:

https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds

could we use this instead of the fabric stuff?

the package now gets uploaded to debian.koumbit.net unstable when it is built properly, both the official package and the automatic build, although it seems that the upstream archive isn't accepting the snapshot builds for some reason. note that the jenkins build is sending too many .changes files (it should send only the last) so that creates checksum errors:

sh-3.2$ /usr/bin/reprepro  -b /srv/reprepro processincoming incoming
File "pool/main/a/aegir-provision/aegir-provision_1.1.orig.tar.gz" is already registered with different checksums!
md5 expected: 6b2111b3c9ca37f9e7c65986633a595a, got: 0afaff905ce24eca8bcdc73fc4412d64
sha1 expected: 8757002a3725ef6dcaffef243b4e84ec0e4634e0, got: 2b3b00b4cee58165c680fbec24f3561e23c43640
sha256 expected: b93e52a822ccb5d61ddfde39769816a1d4176e0c8ba06a9267179483ccd20e9a, got: f0f7da60ab7263ea8eff4e3f46c05bf510de68b348a69fea4828f5d2a3248410
size expected: 95487, got: 106769

next step is to provision a server in the cloud and install the build.

Okay, so now the auto builder also gets uploaded properly. I have uploaded a new 1.1-5 package and immediately migrated it to the testing repository so that precious package is kept even if new auto packages are uploaded to unstable.

I also fixed our debian repository to immediately process new packages so that there is no delay and we can pull packages in right after the build, if the upload works, of course, which is something we can't check unless we carry the version we built to sub-jobs and force that version during apt-get install...

But yeah, that's still the next step: install the built package using apt-get in a new cloudy server thingy.

Alright, so the issue here is that the build script builds from testing. To test the snapshot release, you need unstable.

I tried to fix this in the script but then the build failed... Investigating.

I have refactored into our debian package build the improvements I have done on the Drush package build in Jenkins - it now checks if the upload succeeded and cleans up after itself much better.

Alright!!! That actually *was* the root of the problem - the uploads were failing!

I think this is done people!!! Way to go everyone!

The only thing left here is to test *upgrades* but that's a so big pain in the butt that I'll open a separate issue about it. :P