At first I thought this was due to errors an incorrectly upgraded plug-in, but even new jobs using the bare minimum of plugins (Vagrant, IRC) sometimes disappear after a reboot.

Comments

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
commented

Also note: #2078533: ci.aegirproject.org cannot send notifications to #aegir

anarcat’s picture

anarcat commented

i fixed the IRC issue it seems, see that other issue for details.

now it seems that jenkins was running as root (ouch!). i restarted it as the jenkins user, but had to do the following:

chown -R jenkins /var/log/jenkins/ /var/run/jenkins/ /var/cache/jenkins/war/ /var/lib/jenkins
ulimit -s unlimited

and start jenkins with:

/bin/su -l jenkins --shell=/bin/bash -c /usr/bin/daemon --foreground --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Xmx2048m -XX:MaxPermSize=2048M -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=-1

without -Xmx2048m -XX:MaxPermSize=2048M i was getting memory errors.

the memory usage of jenkins is ridiculous:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14946 jenkins   20   0 4512m 341m  17m S    0  4.3   0:34.40 java

so now i can start jenkins (!?).

but the problem with older jobs being lost remains, and this is why:

Sep 18, 2013 2:36:33 PM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading job D aegir 6.x-1.x install
hudson.util.IOException2: Unable to read /var/lib/jenkins/jobs/D aegir 6.x-1.x install/config.xml
        at hudson.XmlFile.read(XmlFile.java:147)
        at hudson.model.Items.load(Items.java:220)
        at jenkins.model.Jenkins$17.run(Jenkins.java:2554)
        at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
        at jenkins.model.Jenkins$7.runTask(Jenkins.java:888)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
Caused by: com.thoughtworks.xstream.converters.ConversionException: Undefined class: Java::HudsonModel::StreamBuildListener : Undefined class: Java::HudsonModel::StreamBuildListener
---- Debugging information ----
message             : Undefined class: Java::HudsonModel::StreamBuildListener
cause-exception     : java.lang.IllegalArgumentException
cause-message       : Undefined class: Java::HudsonModel::StreamBuildListener
class               : org.jruby.runtime.builtin.IRubyObject
required-type       : org.jruby.runtime.builtin.IRubyObject
converter-type      : org.jenkinsci.jruby.JRubyXStreamConverter
path                : /project/builders/ruby-proxy-object/ruby-object/object/vagrant/host/ui/listener/native
line number         : 202
class[1]            : org.jenkinsci.jruby.JRubyMapper$DynamicProxy
converter-type[1]   : org.jenkinsci.jruby.JavaProxyConverter
class[2]            : hudson.util.DescribableList
converter-type[2]   : hudson.util.DescribableList$ConverterImpl
class[3]            : hudson.model.FreeStyleProject
converter-type[3]   : hudson.util.RobustReflectionConverter
version             : null
-------------------------------
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:111)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:111)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:111)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:111)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:111)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:111)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at org.jenkinsci.jruby.JavaProxyConverter.unmarshal(JavaProxyConverter.java:60)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
        at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:193)
        at hudson.util.DescribableList$ConverterImpl.unmarshal(DescribableList.java:263)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:333)
        at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:275)
        at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061)
        at hudson.util.XStream2.unmarshal(XStream2.java:109)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045)
        at com.thoughtworks.xstream.XStream.fromXML(XStream.java:925)
        at hudson.XmlFile.read(XmlFile.java:143)
        ... 10 more
Caused by: java.lang.IllegalArgumentException: Undefined class: Java::HudsonModel::StreamBuildListener
        at org.jenkinsci.jruby.JRubyXStreamConverter.unmarshal(JRubyXStreamConverter.java:86)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
        ... 68 more
Sep 18, 2013 2:36:34 PM jenkins.InitReactorRunner$1 onAttained

i.e. i have no freaking clue why.

anarcat’s picture

anarcat commented

it turns out that the vagrant plugin is messed up: we had the 1.5-SNAPSHOT plugin installed, but that plugin isn't actually available from jenkins-ci.org. https://wiki.jenkins-ci.org/display/JENKINS/Vagrant+Plugin# -> says that the latest is 0.1.4. if you look at their github (https://github.com/rtyler/vagrant-plugin) however, you see that they are, in their minds at 0.1.6, but that never got pushed to the central repo, it seems.

now we have the choice, either we learn how to build and use jenkins / ruby plugins, or we ditch that plugin and do things by hand.

anarcat’s picture

anarcat commented
Status: Active » Needs work

so it turns out jenkins was automatically upgraded during a nightly upgrade, which broke the vagrant plugin.

i opened an issue upstream in that plugin: https://github.com/rtyler/vagrant-plugin/issues/35

but in the meantime the only solution we've found is to run jenkins by hand.

anarcat’s picture

anarcat commented

disabled the following jobs:

root@zeus:/var/lib/jenkins# grep 'Failed Loading job' /var/log/jenkins/jenkins.log | sed 's/SEVERE: Failed Loading job //' | sort -u | sed 's/[^A-Za-z0-9_ \.-]//g' | while read job; do echo "mv jobs/\"$job\" broken_job/"; done
mv jobs/"D aegir2 apt install" broken_job/
mv jobs/"D aegir 6.x-1.x install" broken_job/
mv jobs/"D aegir 6.x-2.x deb install" broken_job/
mv jobs/"D aegir 6.x-2.x install" broken_job/
mv jobs/"D aegir apt install" broken_job/
mv jobs/"Matrix test" broken_job/
mv jobs/"P Aegir Puppet module smoke test" broken_job/
mv jobs/"P Aegir_up aegir2-dev provision tests" broken_job/
mv jobs/"P Aegir_up aegir2 install test" broken_job/
mv jobs/"P Aegir_up aegir2 provision tests" broken_job/
mv jobs/"P Aegir_up aegir2-unstable provision tests" broken_job/
mv jobs/"P Aegir_up Puppet module smoke test" broken_job/
mv jobs/"P Drush Puppet module smoke test" broken_job/
mv jobs/"P Drush_vagrant Puppet module smoke test" broken_job/
mv jobs/"S aegir 6.x-1.6 install" broken_job/
mv jobs/"S_aegir_6.x-1.x_install" broken_job/
mv jobs/"S_aegir_6.x-2.x_install" broken_job/
mv jobs/"S_aegir_official_deb_install" broken_job/

those were already disabled, i think:

mv: cannot stat `jobs/P Aegir Puppet module smoke test': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2-dev provision tests': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2 install test': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2 provision tests': No such file or directory
mv: cannot stat `jobs/P Aegir_up aegir2-unstable provision tests': No such file or directory
mv: cannot stat `jobs/P Aegir_up Puppet module smoke test': No such file or directory
mv: cannot stat `jobs/P Drush Puppet module smoke test': No such file or directory
mv: cannot stat `jobs/P Drush_vagrant Puppet module smoke test': No such file or directory
anarcat’s picture

anarcat commented

what's up with this?

from what i remember, the next step was to recreate those tasks by hand without using the vagrant plugin... on which there is no update, btw.

helmo’s picture

helmo commented

The repo and issue mentioned in #4seem to have little to no activity.

There are some forks though: https://github.com/rtyler/vagrant-plugin/network Anything usefull there?

Or could we do without the vagrant plugin, like: http://larrycaiyu.com/blog/2011/10/21/make-ci-easier-with-jenkins-ci-and...

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
commented
Status: Needs work » Fixed

This appears to be fixed. Also, we're rebuilding our tests to be as self-contained as possible, in order to make recovery from anything like this much quicker.

We can follow up in #1912248: cleanup jenkins build scripts.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.