If you are lucky that might mask the real issue, but I doubt it… that is an insane number of compaction tasks and indicative of another problem. I would check release notes of 2.0.6+, if I recall that was not a stable version and may have had leaks.
Aside from that, just FYI, if you use native_objects for memtables you can run CMS fine even up to largish (20-30G) heap sizes without long GC pauses (at least over months) . That said look for promotion failures of 131074 DWORDS. Not sure what your GC logging options are, but what you posted doesn’t say why a full GC happened which would be totally tunable, but as I say I don’t think GC is actually your problem > On Dec 31, 2015, at 10:16 AM, Ipremyadav <ipremya...@gmail.com> wrote: > > Simplest option is to use java 8 with G1 gc. > > On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2...@gmail.com > <mailto:chenatu2...@gmail.com>> wrote: > >> I have a cassandra 2.0.6 cluster with four nodes as backup database. The >> only operation is posting data into db. Recently, the full gc of the nodes >> increases apparently and blocks cluster operation. >> >> The load of each node is 10G. The heap is 8G each with default jvm memory >> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost >> default. >> >> For debug, all the clients are disconnected, however the gc is still high. >> >> The GC is too often that it appears in every minute and blocks for nealy 30s. >> >> How to debug this situation >> >> The following is the snippets of gc log: >> >> before full gc histogram >> >> >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs] >> Before GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> Before GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> ---------------------------------------------- >> 1: 61030245 1952967840 java.util.concurrent.FutureTask >> 2: 61031398 1464753552 >> java.util.concurrent.Executors$RunnableAdapter >> 3: 61030312 1464727488 >> java.util.concurrent.LinkedBlockingQueue$Node >> 4: 61030244 1464725856 >> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask >> 5: 214181 13386992 [B >> 6: 202890 9738720 java.nio.HeapByteBuffer >> 7: 37622 5994808 [C >> 8: 42163 5722096 <constMethodKlass> >> 9: 42163 5407152 <methodKlass> >> 10: 4170 4648240 <constantPoolKlass> >> 11: 100060 4002400 >> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo >> 12: 4170 2860816 <instanceKlassKlass> >> 13: 3664 2702720 <constantPoolCacheKlass> >> 14: 4817 2701576 [J >> 15: 2575 1056600 <methodDataKlass> >> 16: 37431 898344 java.lang.String >> 17: 2585 627352 [Ljava.lang.Object; >> 18: 17728 567296 >> java.util.concurrent.ConcurrentHashMap$HashEntry >> 19: 22111 530664 javax.management.ObjectName$Property >> 20: 7595 463824 [S >> 21: 4534 433560 java.lang.Class >> 22: 11581 370592 java.util.HashMap$Entry >> >> ... >> Total 289942930 8399196504 >> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs] >> CMS: Large block 0x00000007f7da9588 >> >> After GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 6335823 >> Max Chunk Size: 6335823 >> Number of Blocks: 1 >> Av. Block Size: 6335823 >> Tree Height: 1 >> After GC: >> Statistics for BinaryTreeDictionary: >> ------------------------------------ >> Total Free Space: 0 >> Max Chunk Size: 0 >> Number of Blocks: 0 >> Tree Height: 0 >> >> >> It seems related to objects of Futuretask. >> >> -- >> 陈硕 Shuo Chen >> chenatu2...@gmail.com <mailto:chenatu2...@gmail.com> >> chens...@whaty.com <mailto:chens...@whaty.com>
smime.p7s
Description: S/MIME cryptographic signature
