Found it. Lots of "out of memory" errors repeated increasingly quickly. Mostly PermGen space. Sort of a melt-down with lots of build tasks and tasks talking to the slaves starting and dying quickly until the whole Tomcat task gave up the ghost.
The monitoring plugin doesn't show any growth in memory usage leading up to this. Does anyone know how accurate those graphs would be? Any ideas on how to figure out whether it just needs more memory or something is eating (slowly or rapidly) itself out of memory space? Lee On Wed, Feb 8, 2012 at 4:25 PM, Sami Tikka <sjti...@gmail.com> wrote: > Look in the tomcat installation dir/logs/catalina.out > > And it looks like you need to start java with higher permgen space than > the default. I myself use jconsole to monitor the java JVM and it's memory > consumption. > > Google will tell you the specifics > > -- Sami > > Lee Meador <l...@leemeador.com> kirjoitti 8.2.2012 kello 20.24: > > > Jenkins 1.446 running under Tomcat 6.0.32, Sun/Oracle Java 1.6 64bit on > RHEL 5.6 Linux 64 bit. That's all running an a VSphere VM with two cpus and > 6 GB memory. > > > > The Tomcat task dies now and then. For example, it died this morning > around 4:15 (based on when the monitoring plugin quit showing activity). > When I came in "ps -ef" didn't have any lines with "java" in them. No java > task running means Tomcat completely died. > > > > It could be coincidence or there could have been something else changed > about that time, but it seems to have originated when I upgraded the > Jenkins version. The old version was1.387 which had been running since > about the first of last March. The new version is 1.446. > > > > Are there various logs that would show what was happening at that time? > If there was an error logged where would I find that error message? If > there was a heap dump or error trace, where would it be? > > > > I need some help figuring out where to start looking. > > > > (I also should mention another problem symptom that may or may not be > related. The builds sometimes start dying with PerGen errors and the web ui > gets slow and then quits responding. In this case, I haven't seen the java > task die. The Tomcat task usually sits at 75-100% of one cpu usage. I've > not seen it go over 100% of that CPU. (This is in 'top') The other tasks > (for Maven or Ant builds and JUnit test runs) seem to run normally and > eventually terminate after a while. I haven't seen the Tomcat task die but > maybe I didn't wait long enough.) > > > > Lee > -- -- Lee Meador Sent from gmail. My real email address is lee AT leemeador.com
