Hello James, > -----Ursprüngliche Nachricht----- > Von: James H. H. Lampert <jam...@touchtonecorp.com.INVALID> > Gesendet: Montag, 6. Februar 2023 18:18 > An: Tomcat Users List <users@tomcat.apache.org> > Betreff: Re: AW: Having trouble with Tomcat crashes. Interesting memory > numbers in Manager > > Thanks, Herr Hoffmann. Your questions were most helpful in determining > what information to gather and share. And thanks in advance to anybody > else who has any insights. > > First, I will note that the seemingly non-sequitur nursery-survivor numbers > aren't just what we see during a crash; they're what we see when it's running > normally. > > On 2/4/23 6:13 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > Could you describe "crash" in a bit more detail? > > Typically, the signed-on users start to get degraded response times, before it > becomes completely unresponsive. > > > - does the tomcat / java process run but is unresponsive? > > Yes. Exactly. And shutting it down (and therefore freeing up the port for a > restart) takes a fairly sizeable amount of time, and leaves a core dump of > approximately 6G size, a Javacore dump of approximately 4M size, and a JIT > dump of approximately 20M size. > > > - does the java process crash itself (then there should be a logfile > > written)? > The job does not generally terminate itself, or even respond to a shutdown > request; it has to be forcibly terminated (given that it's running on an > AS/400, > this would typically be either from WRKACTJOB, or from an ENDJOB > command, or from their GUI console equivalents). > > This may be relevant: even when it is not in this state, the Tomcat server, > when being shut down, tends not to respond readily to shutdown requests. > > > - Is there any OOM message in the logfiles? > Not out-of-memory, but there are chronic problems with contacting outside > web services (many of them involving Oauth2), and with BIRT reporting. > > Around the time of the shutdown, I typically see stuff like: > Unhandled exception > Type=Segmentation error vmState=0x00000000 > J9Generic_Signal_Number=00000004 Signal_Number=0000000b > Error_Value=00000000 Signal_Code=00000032 > > I am not sure whether this is going into catalina.out before or after the job > is > forcibly terminated. > > > - Is the process still alive but CPU at 100% ? > Yes. > > We just had a near-miss as I was typing this: CPU pushing up into the high > 80s, and the JVM job for Tomcat eating up most of it, but it backed down to > something more normal without my having to intervene, and without any > sign of anybody else intervening. > > One of my colleagues managed to get into manager during the near-miss, > and took a screen-shot. The "nursery-allocate" Used was at 400.97M (34%), > "nursery-survivor" as I described last week, "tenured-LOA" Used was at zero > used, and "tenured-SOA" was showing Initial 2918.40M, Total 3648.00M, > Maximum 4864.00M, and Used 1997.72M (41%). > > -- > JHHL > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org
The observations looks like java is running out of memory and the garbage collector can't keep up with making memory free again. Either the GC uses 100% or the application has some cpu intensive procedures. I would guess, that it’s the GC. One option would be to open a JMX port on tomcat and use VisualVM to connect to the java process and inspect the memory and GC usage. When the CPU is eating 100% CPU you might also consider generating a thread dump (kill -3) and check if there are any suspicious threads running. Also setting the java options HeapDumpOnOutOfMemoryError and HeapDumpPath might help, if the process stops because of OOM. If the GC can always free some bytes again, which the application is instantly eating again, an OOM might not occur. You can also add parameters to log some GC statistics, but I never used that : https://sematext.com/blog/java-garbage-collection-logs/ Greetings, Thomas
