What's the size of young generation (-Xmn) ? On Fri, Nov 13, 2015 at 6:38 AM, Jason Wee <peich...@gmail.com> wrote:
> Used to manage/develop for cassandra 1.0.8 for quite sometime. Although > 1.0 was rocking stable but we encountered various problems as load per node > grow beyond 500gb. upgrading is one of the solution but may not be the > solution for you but I strongly recommend you upgrade to 1.1 or 1.2. we > upgraded the java on the cassandra node and cassandra to 1.1 and a lot of > problems went away. > > As for your use cases, a quick solution would probably to just add nodes, > or study client reading pattern so not on a node hot row (the has on the > key), or the client configuration on your application and/or the keyspace > replication. > > hth, > > jason > > On Fri, Nov 13, 2015 at 2:35 PM, Shuo Chen <chenatu2...@gmail.com> wrote: > >> Hi, >> >> We have a small cassandra cluster with 4 nodes for production. All the >> nodes have similar hardware configuration and similar data load. The C* >> version is 1.0.7 (prretty old) >> >> One of the node has much higher cpu usage than others and high full gc >> frequency, but the io of this node is not high and data load of this node >> is even lower. So I have several questions: >> >> 1. Is that normal that one of the node having much higher full gc with >> same jvm configuration? >> 2. Does this node need special gc tuning and how? >> 3. How to find the cause of the full gc? >> >> Thank you guys! >> >> >> The heap size is 8G and max heap size is 16G. The gc config of >> cassandra-env.sh is default: >> >> JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" >> JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" >> JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" >> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" >> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1" >> JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=75" >> JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly" >> >> --------------------- >> I print instance in the gc log: >> >> num #instances #bytes class name >> ---------------------------------------------- >> 1: 2982796 238731200 [B >> 2: 3889672 186704256 java.nio.HeapByteBuffer >> 3: 1749589 55986848 org.apache.cassandra.db.Column >> 4: 1803900 43293600 >> java.util.concurrent.ConcurrentSkipListMap$Node >> 5: 859496 20627904 >> java.util.concurrent.ConcurrentSkipListMap$Index >> 6: 5568 18827912 [J >> 7: 162630 6505200 java.math.BigInteger >> 8: 167572 5716976 [I >> 9: 141698 4534336 >> java.util.concurrent.ConcurrentHashMap$HashEntry >> 10: 141505 4528160 >> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node >> 11: 31491 4376976 <constMethodKlass> >> 12: 31491 4291992 <methodKlass> >> 13: 171695 4120680 org.apache.cassandra.db.DecoratedKey >> 14: 3157 3436120 <constantPoolKlass> >> 15: 141784 3402816 java.lang.Long >> 16: 141624 3398976 org.apache.cassandra.utils.Pair >> 17: 141505 3396120 >> com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue >> 18: 49604 2675352 <symbolKlass> >> 19: 162254 2596064 >> org.apache.cassandra.dht.BigIntegerToken >> ...... >> Total 13337798 641834360 >> >> >> ---------------------- >> The gc part and thread status of system log: >> >> INFO [ScheduledTasks:1] 2015-11-13 14:22:08,681 GCInspector.java (line >> 123) GC for ParNew: 1015 ms for 2 collections, 3886753520 used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:09,683 GCInspector.java (line >> 123) GC for ParNew: 500 ms for 1 collections, 4956287408 used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:10,685 GCInspector.java (line >> 123) GC for ParNew: 627 ms for 1 collections, 5615882296used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:12,015 GCInspector.java (line >> 123) GC for ParNew: 988 ms for 2 collections, 4943363480 used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:13,016 GCInspector.java (line >> 123) GC for ParNew: 373 ms for 1 collections, 5978572832 used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:14,020 GCInspector.java (line >> 123) GC for ParNew: 486 ms for 1 collections, 6209638280used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:15,412 GCInspector.java (line >> 123) GC for ParNew: 898 ms for 2 collections, 6045603728used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:16,413 GCInspector.java (line >> 123) GC for ParNew: 503 ms for 1 collections, 6991263984 used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:17,416 GCInspector.java (line >> 123) GC for ParNew: 746 ms for 1 collections, 7073467384used; max is >> 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,363 GCInspector.java (line >> 123) GC for ConcurrentMarkSweep: 843 ms for 2 collections, 1130423160 used; >> max is 8231321600 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,364 MessagingService.java >> (line 603) 4198 READ messages dropped in last 5000ms >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,364 StatusLogger.java (line >> 50) Pool Name Active Pending Blocked >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,368 StatusLogger.java (line >> 65) ReadStage 32 450 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,370 StatusLogger.java (line >> 65) RequestResponseStage 0 18 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,371 StatusLogger.java (line >> 65) ReadRepairStage 0 3 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,372 StatusLogger.java (line >> 65) MutationStage 2 343 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,373 StatusLogger.java (line >> 65) ReplicateOnWriteStage 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,374 StatusLogger.java (line >> 65) GossipStage 0 3 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,375 StatusLogger.java (line >> 65) AntiEntropyStage 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,396 StatusLogger.java (line >> 65) MigrationStage 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line >> 65) StreamStage 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line >> 65) MemtablePostFlusher 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,397 StatusLogger.java (line >> 65) FlushWriter 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,398 StatusLogger.java (line >> 65) MiscStage 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,398 StatusLogger.java (line >> 65) InternalResponseStage 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,434 StatusLogger.java (line >> 65) HintedHandoff 0 0 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,435 StatusLogger.java (line >> 69) CompactionManager n/a 0 >> INFO [ScheduledTasks:1] 2015-11-13 14:22:33,436 StatusLogger.java (line >> 81) MessagingService n/a 0,67 >> >> -- >> *陈硕* *Shuo Chen* >> chenatu2...@gmail.com >> chens...@whaty.com >> > >
