ci.aegirproject.org loses jobs after every reboot [#2078531]

At first I thought this was due to errors an incorrectly upgraded plug-in, but even new jobs using the bare minimum of plugins (Vagrant, IRC) sometimes disappear after a reboot.

Comments

Comment #1

ergonlogic

he/him

English

Montréal, Québec 🇨🇦

CreditAttribution: ergonlogic commented 31 August 2013 at 17:59

Also note: #2078533: ci.aegirproject.org cannot send notifications to #aegir

Comment #3

anarcat CreditAttribution: anarcat commented 18 September 2013 at 19:19

it turns out that the vagrant plugin is messed up: we had the 1.5-SNAPSHOT plugin installed, but that plugin isn't actually available from jenkins-ci.org. https://wiki.jenkins-ci.org/display/JENKINS/Vagrant+Plugin# -> says that the latest is 0.1.4. if you look at their github (https://github.com/rtyler/vagrant-plugin) however, you see that they are, in their minds at 0.1.6, but that never got pushed to the central repo, it seems.

now we have the choice, either we learn how to build and use jenkins / ruby plugins, or we ditch that plugin and do things by hand.

Comment #4

anarcat CreditAttribution: anarcat commented 18 September 2013 at 19:31

Status:

Active

» Needs work

so it turns out jenkins was automatically upgraded during a nightly upgrade, which broke the vagrant plugin.

i opened an issue upstream in that plugin: https://github.com/rtyler/vagrant-plugin/issues/35

but in the meantime the only solution we've found is to run jenkins by hand.

Comment #5

anarcat CreditAttribution: anarcat commented 18 September 2013 at 19:36

disabled the following jobs:

root@zeus:/var/lib/jenkins# grep 'Failed Loading job' /var/log/jenkins/jenkins.log | sed 's/SEVERE: Failed Loading job //' | sort -u | sed 's/[^A-Za-z0-9_ \.-]//g' | while read job; do echo "mv jobs/\"$job\" broken_job/"; done
mv jobs/"D aegir2 apt install" broken_job/
mv jobs/"D aegir 6.x-1.x install" broken_job/
mv jobs/"D aegir 6.x-2.x deb install" broken_job/
mv jobs/"D aegir 6.x-2.x install" broken_job/
mv jobs/"D aegir apt install" broken_job/
mv jobs/"Matrix test" broken_job/
mv jobs/"P Aegir Puppet module smoke test" broken_job/
mv jobs/"P Aegir_up aegir2-dev provision tests" broken_job/
mv jobs/"P Aegir_up aegir2 install test" broken_job/
mv jobs/"P Aegir_up aegir2 provision tests" broken_job/
mv jobs/"P Aegir_up aegir2-unstable provision tests" broken_job/
mv jobs/"P Aegir_up Puppet module smoke test" broken_job/
mv jobs/"P Drush Puppet module smoke test" broken_job/
mv jobs/"P Drush_vagrant Puppet module smoke test" broken_job/
mv jobs/"S aegir 6.x-1.6 install" broken_job/
mv jobs/"S_aegir_6.x-1.x_install" broken_job/
mv jobs/"S_aegir_6.x-2.x_install" broken_job/
mv jobs/"S_aegir_official_deb_install" broken_job/

those were already disabled, i think:

mv: cannot stat `jobs/P Aegir Puppet module smoke test': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2-dev provision tests': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2 install test': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2 provision tests': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2-unstable provision tests': No such file or directory
mv: cannot stat `jobs/P Aegir_up Puppet module smoke test': No such file or directory
mv: cannot stat `jobs/P Drush Puppet module smoke test': No such file or directory
mv: cannot stat `jobs/P Drush_vagrant Puppet module smoke test': No such file or directory

Comment #6

anarcat CreditAttribution: anarcat commented 24 September 2013 at 15:12

what's up with this?

from what i remember, the next step was to recreate those tasks by hand without using the vagrant plugin... on which there is no update, btw.

Comment #7

helmo CreditAttribution: helmo commented 10 October 2013 at 09:41

The repo and issue mentioned in #4seem to have little to no activity.

There are some forks though: https://github.com/rtyler/vagrant-plugin/network Anything usefull there?

Or could we do without the vagrant plugin, like: http://larrycaiyu.com/blog/2011/10/21/make-ci-easier-with-jenkins-ci-and...

Comment #8

ergonlogic

he/him

English

Montréal, Québec 🇨🇦

CreditAttribution: ergonlogic commented 18 October 2013 at 13:19

Status:

Needs work

» Fixed

This appears to be fixed. Also, we're rebuilding our tests to be as self-contained as possible, in order to make recovery from anything like this much quicker.

We can follow up in #1912248: cleanup jenkins build scripts.