Hello Community,
Am troubleshooting an issue with sudden and sustained high CPU on nodes in a 3
node cluster. This takes place when there is minimal load on the servers, and
continues indefinitely until I stop and restart a node. All nodes (3) seem to
be effected by the same issue, however it doesn't occur simultaneous on them
(although all potentially could be effected at the same time).
Here's what I've been doing to troubleshoot.
* vmstat - have run on all servers and there is *no* swap going on
* iostat - have run on all servers and there is no indication that the disk is
in anyway a bottleneck
* nodetool - nothing is backed up. Not using all available heap (about 2/3rds).
* Have tracked it down to a specific thread (top -H). When I find that
thread's hex equivalent in a jstack dump I see the following
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000001d48800
nid=0x4534 runnable
Which makes me think that perhaps it's something to do with GC configuration.
Unfortunately I'm not able to determine why this might be happening. Any
suggestions on how to continue troubleshooting?
btw - here's my jvm (just in case)
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
Thanks!
Matt
