Thank you to the several people who made suggestions, either on the list
or by direct email!
I think I have solved the problem by simply using the -server option to
java when starting tomcat. By default tomcat uses the default VM, which
for Sun JDK1.5 is the client VM. By default the client VM sets a heap
size limit of 64MB. It seems that as I have added jsp pages and webapps
the total memory footprint of the webapps on my server had crept up over
time to more than 64MB so the virtual machine was "thrashing" in garbage
collection. I have plenty of memory so the higher limit makes complete
sense for me. The tool that led me to this conclusion was JConsole,
which I was unfamiliar with when I first posted. JConsole comes with the
jdk and allows you to connect to the VM and monitor things like memory
usage and limits and garbage collection:
http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html
It was due to JConsole that I realized I was using the client VM and
that memory was limited to 64MB. When I switched to the server VM the
memory limit was set to a much higher default, and after watching usage
for a while I realized that the "steady state" memory usage was over
64MB. JConsole seems to be a very useful tool, and perhaps a mention
could be added to the tomcat documentation someplace under
administration. Thank you to Chuck for suggesting using JConsole.
This page gives really simple instructions on how to set up JConsole
with password protected access:
http://www.opennms.org/index.php/Tomcat_5.5_JMX_How-To
I was a little surprised to see that one could access tomcat passwords
(like the password for manager access) using JConsole, so think twice
before setting it up without authentication!
Thanks again to everyone who helped me out on this!
Lindsay
Lindsay Patten wrote:
I am looking for information on how to debug a problem. Any pointers
to documentation, suggestions of approaches etc. would be much
appreciated.
I am having a problem with my tomcat server periodically going into a
state where it is using up all the available cpu and providing very
slow response. None of the jsp or servlets on the site should take up
a lot of cpu, they all just do relatively simple database queries and
produce simple html pages from the results. During the times when the
cpu is busy the database will generally not be under any significant
load.
Top shows process 324 (0x144) as the busy process, and ps shows it
having consumed a lot of cpu:
PID TTY STAT TIME COMMAND
324 ? S 837:41 /usr/java/jdk1.5.0_10/bin/java -Djava...
Expecting to find that I had some error in my webapp I did a few
thread dumps, but assuming that nid in the thread dump corresponds to
pid from ps I found that the process that was spinning was "VM Thread":
"VM Thread" prio=1 tid=0x08098a48 nid=0x144 runnable
The thread dump does not give any additional information about this
thread. I presume it is the Virtual Machine itself.
If I look at the system status using the Tomcat manager webapp there
are often requests listed with ridiculously large values in the Time
column, several hundred seconds for jsp pages that only take a
fraction of a second to generate, and I can cut an paste the request
into my browser and the request will finish in a fraction of a second.
Sometimes it seems that reloading some of the webapps can "fix" the
problem, at least for a while - I don't know for sure, it could be
coincidence, but if I don't do anything the problem can persist for
long periods of time and then go away after a few webapp reloads. I
have tried to detect a pattern about which app reload solves the
problem but it seems to be different different times. vmstat doesn't
show the machine as thrashing or using much IO (everything basically
fits into physical memory). The only thing I can think of is that the
VM is going into a marathon garbage collection phase, but I don't know
how to test that hypothesis, and it seems odd that even that could go
on for lengthy periods of time.
Help?!?
JVM: jdk1.5.0_10
Tomcat: apache-tomcat-6.0.13
uname -a: Linux [hostname here] 2.4.34-grsec #27 SMP Mon Jan 22
21:19:29 CST 2007 i686 unknown
The system is a two cpu system with 4GB RAM, and when not in this
spinning state it handles its load with 10-15% cpu load.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
