On 29.05.2012 17:03, Kevin wrote:
Hi
We have a cluster of tomcat servers being used on a very high volume
website. We've noticed that for the first 5-6 hours after an application
re-load that Full GC will run every 2 minutes pausing the application for
anywhere between 5 and 20 seconds. After 5-6 hours the full GC will no
longer run until tomcat is reloaded. This behavior is repeatable.
We discounted our EhCache settings as potential issue because tomcat can
be killed without this issue showing up. It's not until we actually
reload the WAR with changes (clearing the work directory) and restart
tomcat that this issue shows up. Traffic level isn't a factor we'll go
through our peak hours with no problem.
The servers are all dual quad core with 32GB of RAM running Centos 5 and
tomcat 6.0.18. We've attempted every suggested GC setting change
suggested on the net and by developers and system administrators. Our
java opts are being played with on a daily basis however here are the
settings that correspond to the sample GC log below:
-server -Xmx27g -Xms27g -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC
-XX:+PrintTenuringDistribution -Dsun.rmi.dgc.client.gcInterval=900000
-Dsun.rmi.dgc.server.gcInterval=900000 -XX:NewSize=8g -XX:SurvivorRatio=16
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
343.376: [Full GC 343.378: [CMS: 1158107K->1312570K(19922944K), 3.4129290
secs] 2884580K->1312570K(27845568K), [CMS Perm : 83964K->47203K(83968K)],
3.4168600 secs] [Times: user=3.87 sys=0.02, real=3.41 secs]
20526.469: [Full GC 20526.475: [CMS: 9175755K->9210800K(19922944K),
33.1161300 secs] 13632232K->9210800K(27845568K), [CMS Perm :
83967K->53332K(83968K)], 33.1254170 secs] [Times: user=33.12 sys=0.02,
real=33.12 secs]
Looks to me as the oerm gen fills up and triggers the Full GC. Then perm
gen is back to only be used by about 60% but rapidly fills up again. So
it seems your are using a lot of classes that are only used for a short
period of time.
Add
-XX:+TraceClassUnloading
which adds the names of the classes which are unloaded to the GC output.
Then check, whether the names of the classes give you an idea, why you
load and unload that many classes for a long time (and why it finally
gets stable).
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
