We have a regressions server, running regressions of different applications, using Jenkins.
Some of these applications, tend to misbehave during the regression run, and the build needs to be aborted. The runtimes of these applications are a mix of JVM and native code, interfaced via JNI, and also httpd instances. Now, in our setup, we have the build timeout plugin configured. Typically this is set as follows (with time out action being Abort the build on some jobs and Fail the build on some jobs). <https://lh4.googleusercontent.com/-MpClzRtG9VI/U_74sBO5tlI/AAAAAAAAAHc/3-2GJdmqebU/s1600/2014-08-28_103817.png> Now, what we see is that when a build fails, the JVMs and servers and httpd instances corresponding to the build are still up. This keeps happening for a series of jobs, the OS runs out of memory and this crashes the machine. We have confirmed these process have JENKINS_SERVER_COOKIE environment variable set, so jenkins should in principle be able to clean them up. Looking at the documentation for build timeout plugin (https://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin) I see Because Java only allows threads to be interrupted at a set of fixed locations, depending on how a build hangs, the abort operation might not take effect. For example, - if Jenkins is waiting for child processes to complete, it can abort right away. - if Jenkins is stuck in an infinite loop, it can never be aborted. - if Jenkins is doing a network or file I/O within the Java VM (such as lengthy file copy or SVN update), it cannot be aborted. So if you think the build time out isn't taking effect, our default assumption is that the build is hanging at the place that cannot be interrupted. If you suspect otherwise, please obtain the thread dump <https://wiki.jenkins-ci.org/display/JENKINS/Build+is+hanging> and report it. So am planning to collect the thread dump for the next set of runs and understand the issue better. Also I was thinking of enabling more logs for the build timeout plugin. Looking at the configure logging help page, https://wiki.jenkins-ci.org/display/JENKINS/Logging, looks like I need the name of the logger to configure for detailed logging. <https://lh4.googleusercontent.com/-pGK_Ooyuo7Q/U_76XD3ZfoI/AAAAAAAAAHo/98cv_4WPJiA/s1600/2014-08-28_104517.png> Can someone help me with the name of the logger to configure, please? Also any other pointers to debug the issue at hand would be more than welcome ... thanks and regards, pankaj -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
