Hi, We have a small cassandra cluster with 4 nodes for production. All the nodes have similar hardware configuration and similar data load. The C* version is 1.0.7 (prretty old)
One of the node has much higher cpu usage than others and high full gc frequency, but the io of this node is not high and data load of this node is even lower. So I have several questions: 1. Is that normal that one of the node having much higher full gc with same jvm configuration? 2. Does this node need special gc tuning and how? 3. How to find the cause of the full gc? Thank you guys! The heap size is 8G and max heap size is 16G. The gc config of cassandra-env.sh is default: JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1" JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75" JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly" --------------------- I print instance in the gc log: num #instances #bytes class name ---------------------------------------------- 1: 2982796 238731200 [B 2: 3889672 186704256 java.nio.HeapByteBuffer 3: 1749589 55986848 org.apache.cassandra.db.Column 4: 1803900 43293600 java.util.concurrent.ConcurrentSkipListMap$Node 5: 859496 20627904 java.util.concurrent.ConcurrentSkipListMap$Index 6: 5568 18827912 [J 7: 162630 6505200 java.math.BigInteger 8: 167572 5716976 [I 9: 141698 4534336 java.util.concurrent.ConcurrentHashMap$HashEntry 10: 141505 4528160 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node 11: 31491 4376976 <constMethodKlass> 12: 31491 4291992 <methodKlass> 13: 171695 4120680 org.apache.cassandra.db.DecoratedKey 14: 3157 3436120 <constantPoolKlass> 15: 141784 3402816 java.lang.Long 16: 141624 3398976 org.apache.cassandra.utils.Pair 17: 141505 3396120 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue 18: 49604 2675352 <symbolKlass> 19: 162254 2596064 org.apache.cassandra.dht.BigIntegerToken ...... Total 13337798 641834360 ---------------------- The gc part and thread status of system log: INFO [ScheduledTasks:1] 2015-11-13 14:22:08,681 GCInspector.java (line 123) GC for ParNew: 1015 ms for 2 collections, 3886753520 used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:09,683 GCInspector.java (line 123) GC for ParNew: 500 ms for 1 collections, 4956287408 used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:10,685 GCInspector.java (line 123) GC for ParNew: 627 ms for 1 collections, 5615882296used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:12,015 GCInspector.java (line 123) GC for ParNew: 988 ms for 2 collections, 4943363480 used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:13,016 GCInspector.java (line 123) GC for ParNew: 373 ms for 1 collections, 5978572832 used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:14,020 GCInspector.java (line 123) GC for ParNew: 486 ms for 1 collections, 6209638280used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:15,412 GCInspector.java (line 123) GC for ParNew: 898 ms for 2 collections, 6045603728used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:16,413 GCInspector.java (line 123) GC for ParNew: 503 ms for 1 collections, 6991263984 used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:17,416 GCInspector.java (line 123) GC for ParNew: 746 ms for 1 collections, 7073467384used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,363 GCInspector.java (line 123) GC for ConcurrentMarkSweep: 843 ms for 2 collections, 1130423160 used; max is 8231321600 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,364 MessagingService.java (line 603) 4198 READ messages dropped in last 5000ms INFO [ScheduledTasks:1] 2015-11-13 14:22:33,364 StatusLogger.java (line 50) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2015-11-13 14:22:33,368 StatusLogger.java (line 65) ReadStage 32 450 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,370 StatusLogger.java (line 65) RequestResponseStage 0 18 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,371 StatusLogger.java (line 65) ReadRepairStage 0 3 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,372 StatusLogger.java (line 65) MutationStage 2 343 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,373 StatusLogger.java (line 65) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,374 StatusLogger.java (line 65) GossipStage 0 3 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,375 StatusLogger.java (line 65) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,396 StatusLogger.java (line 65) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line 65) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line 65) MemtablePostFlusher 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line 65) FlushWriter 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,398 StatusLogger.java (line 65) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,398 StatusLogger.java (line 65) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,434 StatusLogger.java (line 65) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,435 StatusLogger.java (line 69) CompactionManager n/a 0 INFO [ScheduledTasks:1] 2015-11-13 14:22:33,436 StatusLogger.java (line 81) MessagingService n/a 0,67 -- *陈硕* *Shuo Chen* chenatu2...@gmail.com chens...@whaty.com
