On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
George,
On 11/15/16 10:46 AM, George I. Develekos wrote:
Hello guys,
We are having problems on a production system with very long "full
GC" times, as long as1200sec real time (!!!).
We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
7.0.64.
Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
No other custom memory-related settings are in place.
Looking at the GC log, the last few Full-GC entries are:
1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
user=1.96 sys=0.13, real=2.11 secs]
2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
[CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
user=6.12 sys=1.36, real=*23.21 secs*]
4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
user=2.18 sys=0.14, real=2.34 secs]
What can we do?
1367.020 Full GC duration=2.11s
2579.317 Full GC duration=1212.49s
So your full GC immediately started another full GC that took 20 minutes
?
Are you only showing certain FULL GC activity from your log, or is
that everything?
CMS should have a mark and then a sweep each time, but your times
don't seem to add up.
also note that the whole point of CMS is that there isn't any
stop-the-world during the mark portion of the process.
Are you actually experiencing a problem, or are you just suffering
from instrumentor's remorse?
- -chris
Chris,
What I listed is the result of the command:
grep "Full GC" gc.log
So (obviously) I have skipped other GC activity, i.e. whatever GC
activity didn't include the "Full GC" string.
Yes we are having app trouble due to the GC delays so this is a real
problem. Our application has real-time constraints so the GC delays
cannot be tolerated. I selected those GC options _in order to avoid
_long GC times.
Additionally, these periods coincide with high CPU for that JVM
process. From 5-20% CPU where it is normally, it jumps to 60% ore more.
Once GC is done, our app rushes to catch up with tasks that had to wait
for GC to finish.
Answering another question from a member who has kindly responded, yes
the server is running other stuff. Basically it runs three tomcats, the
main one being this one. It also runs a DB2 database that has
close-to-zero activity.
George
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
