Simplest option is to use java 8 with G1 gc.
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2...@gmail.com> wrote: > > I have a cassandra 2.0.6 cluster with four nodes as backup database. The only > operation is posting data into db. Recently, the full gc of the nodes > increases apparently and blocks cluster operation. > > The load of each node is 10G. The heap is 8G each with default jvm memory > settings. The cpu is 24 cores. The settings of cassandra.yaml are almost > default. > > For debug, all the clients are disconnected, however the gc is still high. > > The GC is too often that it appears in every minute and blocks for nealy 30s. > > How to debug this situation > > The following is the snippets of gc log: > > before full gc histogram > > > Total Free Space: 0 > Max Chunk Size: 0 > Number of Blocks: 0 > Tree Height: 0 > 1620.992: [GC 7377342K(8178944K), 0.1380260 secs] > Before GC: > Statistics for BinaryTreeDictionary: > ------------------------------------ > Total Free Space: 0 > Max Chunk Size: 0 > Number of Blocks: 0 > Tree Height: 0 > Before GC: > Statistics for BinaryTreeDictionary: > ------------------------------------ > Total Free Space: 0 > Max Chunk Size: 0 > Number of Blocks: 0 > Tree Height: 0 > ---------------------------------------------- > 1: 61030245 1952967840 java.util.concurrent.FutureTask > 2: 61031398 1464753552 > java.util.concurrent.Executors$RunnableAdapter > 3: 61030312 1464727488 > java.util.concurrent.LinkedBlockingQueue$Node > 4: 61030244 1464725856 > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask > 5: 214181 13386992 [B > 6: 202890 9738720 java.nio.HeapByteBuffer > 7: 37622 5994808 [C > 8: 42163 5722096 <constMethodKlass> > 9: 42163 5407152 <methodKlass> > 10: 4170 4648240 <constantPoolKlass> > 11: 100060 4002400 > org.apache.cassandra.io.sstable.IndexHelper$IndexInfo > 12: 4170 2860816 <instanceKlassKlass> > 13: 3664 2702720 <constantPoolCacheKlass> > 14: 4817 2701576 [J > 15: 2575 1056600 <methodDataKlass> > 16: 37431 898344 java.lang.String > 17: 2585 627352 [Ljava.lang.Object; > 18: 17728 567296 > java.util.concurrent.ConcurrentHashMap$HashEntry > 19: 22111 530664 javax.management.ObjectName$Property > 20: 7595 463824 [S > 21: 4534 433560 java.lang.Class > 22: 11581 370592 java.util.HashMap$Entry > > ... > Total 289942930 8399196504 > 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs] > CMS: Large block 0x00000007f7da9588 > > After GC: > Statistics for BinaryTreeDictionary: > ------------------------------------ > Total Free Space: 6335823 > Max Chunk Size: 6335823 > Number of Blocks: 1 > Av. Block Size: 6335823 > Tree Height: 1 > After GC: > Statistics for BinaryTreeDictionary: > ------------------------------------ > Total Free Space: 0 > Max Chunk Size: 0 > Number of Blocks: 0 > Tree Height: 0 > > > It seems related to objects of Futuretask. > > -- > 陈硕 Shuo Chen > chenatu2...@gmail.com > chens...@whaty.com
