it appears that the JVM slows everything down to a crawl, including the code path which should lead to another accept being called., for up to 8 minutes!!!
Furthermore, the mpstat has the nice porperty that CPU usage adds up to exactly 100%, i.e. a single CPU is used... no more, no less. This corresponds to 12% or 13% CPU utilization shown in prstat based on 8 CPUs. My interpretation is that the JVM is effectively preventing parallel execution (which otherwise appears to work fine).
Nearly all threads either wait, read from a Socket, or zip/unzip data.
I'm not sure what all that means, but Tomcat appears to be a victim of it. I'll experiment some more. Main difference with the systems Rainer mentioned is the JVM (1.4.2_04) and the CPU (Sparc III 1.2GHz).
If any of this rings a bell, drop me a note. I'll be happy to share data as appropriate.
I'll repost to the list only if I learn anything which impacts Tomcat directly
(other than that the code path to hand of the socket accept responsibility
is not suitable for _very_ high hit rates, which does not worry me too
much at this point).
Cheers! Martin
Martin Schulz wrote:
Rainer,
Thanks for the tips. I am about to take timing stats internally in the ThreadPool and the Tcp workers. Also, the described symptoms do not disappear, but seem to be of much shorter duration when only 4 CPUs are used for the application. I'll summarize what I find.
Martin
Rainer Jung wrote:
Hi,
we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours!
You should send a signal "QUIT" to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09).
In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is.
Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM?
Sincerely
Rainer Jung
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
