Used to manage/develop for cassandra 1.0.8 for quite sometime. Although 1.0 was rocking stable but we encountered various problems as load per node grow beyond 500gb. upgrading is one of the solution but may not be the solution for you but I strongly recommend you upgrade to 1.1 or 1.2. we upgraded the java on the cassandra node and cassandra to 1.1 and a lot of problems went away.
As for your use cases, a quick solution would probably to just add nodes, or study client reading pattern so not on a node hot row (the has on the key), or the client configuration on your application and/or the keyspace replication. hth, jason On Fri, Nov 13, 2015 at 2:35 PM, Shuo Chen <chenatu2...@gmail.com> wrote: > Hi, > > We have a small cassandra cluster with 4 nodes for production. All the > nodes have similar hardware configuration and similar data load. The C* > version is 1.0.7 (prretty old) > > One of the node has much higher cpu usage than others and high full gc > frequency, but the io of this node is not high and data load of this node > is even lower. So I have several questions: > > 1. Is that normal that one of the node having much higher full gc with > same jvm configuration? > 2. Does this node need special gc tuning and how? > 3. How to find the cause of the full gc? > > Thank you guys! > > > The heap size is 8G and max heap size is 16G. The gc config of > cassandra-env.sh is default: > > JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" > JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" > JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" > JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" > JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1" > JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75" > JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly" > > --------------------- > I print instance in the gc log: > > num #instances #bytes class name > ---------------------------------------------- > 1: 2982796 238731200 [B > 2: 3889672 186704256 java.nio.HeapByteBuffer > 3: 1749589 55986848 org.apache.cassandra.db.Column > 4: 1803900 43293600 > java.util.concurrent.ConcurrentSkipListMap$Node > 5: 859496 20627904 > java.util.concurrent.ConcurrentSkipListMap$Index > 6: 5568 18827912 [J > 7: 162630 6505200 java.math.BigInteger > 8: 167572 5716976 [I > 9: 141698 4534336 > java.util.concurrent.ConcurrentHashMap$HashEntry > 10: 141505 4528160 > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node > 11: 31491 4376976 <constMethodKlass> > 12: 31491 4291992 <methodKlass> > 13: 171695 4120680 org.apache.cassandra.db.DecoratedKey > 14: 3157 3436120 <constantPoolKlass> > 15: 141784 3402816 java.lang.Long > 16: 141624 3398976 org.apache.cassandra.utils.Pair > 17: 141505 3396120 > com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue > 18: 49604 2675352 <symbolKlass> > 19: 162254 2596064 > org.apache.cassandra.dht.BigIntegerToken > ...... > Total 13337798 641834360 > > > ---------------------- > The gc part and thread status of system log: > > INFO [ScheduledTasks:1] 2015-11-13 14:22:08,681 GCInspector.java (line > 123) GC for ParNew: 1015 ms for 2 collections, 3886753520 used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:09,683 GCInspector.java (line > 123) GC for ParNew: 500 ms for 1 collections, 4956287408 used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:10,685 GCInspector.java (line > 123) GC for ParNew: 627 ms for 1 collections, 5615882296used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:12,015 GCInspector.java (line > 123) GC for ParNew: 988 ms for 2 collections, 4943363480 used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:13,016 GCInspector.java (line > 123) GC for ParNew: 373 ms for 1 collections, 5978572832 used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:14,020 GCInspector.java (line > 123) GC for ParNew: 486 ms for 1 collections, 6209638280used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:15,412 GCInspector.java (line > 123) GC for ParNew: 898 ms for 2 collections, 6045603728used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:16,413 GCInspector.java (line > 123) GC for ParNew: 503 ms for 1 collections, 6991263984 used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:17,416 GCInspector.java (line > 123) GC for ParNew: 746 ms for 1 collections, 7073467384used; max is > 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,363 GCInspector.java (line > 123) GC for ConcurrentMarkSweep: 843 ms for 2 collections, 1130423160 used; > max is 8231321600 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,364 MessagingService.java > (line 603) 4198 READ messages dropped in last 5000ms > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,364 StatusLogger.java (line > 50) Pool Name Active Pending Blocked > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,368 StatusLogger.java (line > 65) ReadStage 32 450 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,370 StatusLogger.java (line > 65) RequestResponseStage 0 18 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,371 StatusLogger.java (line > 65) ReadRepairStage 0 3 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,372 StatusLogger.java (line > 65) MutationStage 2 343 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,373 StatusLogger.java (line > 65) ReplicateOnWriteStage 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,374 StatusLogger.java (line > 65) GossipStage 0 3 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,375 StatusLogger.java (line > 65) AntiEntropyStage 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,396 StatusLogger.java (line > 65) MigrationStage 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line > 65) StreamStage 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line > 65) MemtablePostFlusher 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line > 65) FlushWriter 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,398 StatusLogger.java (line > 65) MiscStage 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,398 StatusLogger.java (line > 65) InternalResponseStage 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,434 StatusLogger.java (line > 65) HintedHandoff 0 0 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,435 StatusLogger.java (line > 69) CompactionManager n/a 0 > INFO [ScheduledTasks:1] 2015-11-13 14:22:33,436 StatusLogger.java (line > 81) MessagingService n/a 0,67 > > -- > *陈硕* *Shuo Chen* > chenatu2...@gmail.com > chens...@whaty.com >
