Hey guys, Ofter the last few weeks we had a lot of problems with Jenkins actually reporting build problems that were actually infrastructure related. I try and explain the issues and how we try to prevent them from happening in the future.
The GIT plugin used by jenkins has a build in timeout of 10 minutes, if a git operation takes longer than this it will report an error. This timeout is not configurable from within Jenkins and needs to be set with a system property. I’ve made a patch on the cloud-jenkins plugin that will allow us to set custom system properties for the jclouds slaves. This resolved most of the intermittent build problems. There was also the recent change to java 7 for master. This caused most of the master builds to fail on Jenkins as all slaves were setup to explicitly use java 6 (at that was our target platform). Now we made some changes so each build can specify which JDK version it will use at build time. This works on the cloudstack-buildslaves but not yet on the rpm builders. Because of this the debian and redhat packages are not built yet. There were also some problems with the artifacts required for the awsapi. Due to a configuration in the poms for the transient dependencies (outside our tree) several artifacts would have be downloaded from a maven repository that no longer existed. However the site didn’t return a 404 that maven would pickup, but a 200 with an index page so maven would think it properly downloaded the pom/jar file. Causing grief later in the proces. We now use a custom settings.xml for the master build that redirects this particular repository to the archiva instance that david setup. The positive news is that we now have upgraded the service offering for the cloudstack-buildslaves to 4 x 2.6Ghz cores and 4Gb ram, further reducing the main master build to just under 8 minutes (including awsapi). It’s not completely done yet, but hopefully this will help getting Jenkins back into a stable state where we can rely on failures being real failures in the build and not in the infrastructure. Cheers, Hugo
