Hi,
I am using Cassandra 2.0.16 with 4 nodes and found too many compactions for
this cluster. This caused too much full gc and choked the system. I have
discussed the high gc in previous mails but didnot get the satisfied
answers.
To clarify the source of the compactions, I shutdown all the clients and
there is no reading and writing requests outside. Besides,
hinted_handoff_enabled: false is set. The cluster is restarted yesterday
and have run for 1.5 days.
The heap is 8G and all the compaction settings are default settings. I
printed parts of gc histogram as follows:
num #instances #bytes class name
----------------------------------------------
1: 67758530 2168272960 java.util.concurrent.FutureTask
2: 67759745 1626233880
java.util.concurrent.Executors$RunnableAdapter
3: 67758576 1626205824
java.util.concurrent.LinkedBlockingQueue$Node
4: 67758529 1626204696
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
5: 16935 72995576 [B
6: 240534 11545632 java.nio.HeapByteBuffer
7: 37416 5969800 [C
8: 41447 5624856 <constMethodKlass>
9: 41447 5315504 <methodKlass>
10: 104850 5032800
edu.stanford.ppl.concurrent.SnapTreeMap$Node
11: 4110 4564144 <constantPoolKlass>
12: 104781 3352992 org.apache.cassandra.db.Column
13: 4110 2824016 <instanceKlassKlass>
14: 4850 2689568 [J
15: 3564 2634240 <constantPoolCacheKlass>
16: 3016 1252544 <methodDataKlass>
17: 37192 892608 java.lang.String
18: 17641 564512
java.util.concurrent.ConcurrentHashMap$HashEntry
As we can see in this list, objects of compaction tasks remain in the heap.
The nodetool compactionstats as follows:
pending tasks: 64637410
Active compaction remaining time : n/a
Parts of nodetool compactionhistory:
Compaction History:
id keyspace_name
columnfamily_name compacted_at bytes_in
bytes_out rows_merged
8e4f8830-b04f-11e5-a211-45b7aa88107c system
sstable_activity 1451629144115 3342 915
{4:23}
96a6fcb0-b04b-11e5-a211-45b7aa88107c system hints
1451627440123 18970740 18970740 {1:1}
aefc3f10-b1b2-11e5-a211-45b7aa88107c system
sstable_activity 1451781670273 3201 906
{4:23}
1e76f1b0-b180-11e5-a211-45b7aa88107c system
sstable_activity 1451759952971 3303 700
{4:23}
e4d7d220-b518-11e5-ac1a-45b7aa88107c system hints
1452155422786 7837400 7746243 {1:1}
f305ddd0-b52a-11e5-bce5-45b7aa88107c system hints
1452163177517 5509472 0 {3:1}
5c5b15b0-b581-11e5-bce5-45b7aa88107c system
sstable_activity 1452200290955 3387 866
{4:23}
9cc8af70-b24a-11e5-a211-45b7aa88107c system
sstable_activity 1451846923239 3134 773
{4:23}
cf9e9400-b439-11e5-ac1a-45b7aa88107c system hints
1452059609408 10524457 10490132 {3:1}
ea1b5140-b2e2-11e5-a211-45b7aa88107c system
sstable_activity 1451912336468 3243 804
{4:23}
a4b41910-b2b1-11e5-a211-45b7aa88107c system hints
1451891174689 15426047 15236142 {1:3}
f1bb37a0-b204-11e5-a211-45b7aa88107c system hints
1451817000986 17713746 17566349 {1:1}
fcb5bd60-b4d9-11e5-ac1a-45b7aa88107c system
sstable_activity 1452128404534 3146 775
{4:22}
dcd0c720-b0b4-11e5-a211-45b7aa88107c system
sstable_activity 1451672654993 3189 797
{4:23}
ba0f2d60-b50c-11e5-ac1a-45b7aa88107c system
sstable_activity 1452150197046 3055 739
{4:22}
a0681d00-b3ad-11e5-a211-45b7aa88107c system
sstable_activity 1451999400656 3082 770
{4:23}
ce6b4d90-b35c-11e5-a211-45b7aa88107c system hints
1451964688617 13546056 13363986 {1:2}
a17d8f80-b24b-11e5-a211-45b7aa88107c system hints
1451847360632 16746317 16600978 {1:1}
1549d4a0-b51c-11e5-bce5-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1452156792554
46077372 41485236 {1:9, 2:4, 3:2, 4:5}
68e064a0-b439-11e5-ac1a-45b7aa88107c system local
1452059437034 1255 680 {4:1}
38591c10-b22b-11e5-a211-45b7aa88107c system hints
1451833440337 17351684 17002773 {1:1}
aba057f0-b1b4-11e5-a211-45b7aa88107c system hints
1451782523631 18743329 18577220 {1:1}
1fe29f40-b568-11e5-bce5-45b7aa88107c system
sstable_activity 1452189452084 3475 823
{4:23}
f7e02540-b1bf-11e5-a211-45b7aa88107c system hints
1451787376020 18577220 18403648 {1:1}
b3526be0-b3f9-11e5-a211-45b7aa88107c system
sstable_activity 1452032074142 3344 795
{1:2, 4:25}
2ffde460-b45b-11e5-ac1a-45b7aa88107c system
sstable_activity 1452073944486 3318 835
{4:22}
508ed070-b3fe-11e5-a211-45b7aa88107c system hints
1452034055927 11448413 11309063 {1:2}
fdbf8750-b44c-11e5-ac1a-45b7aa88107c system hints
1452067847237 10434789 10400837 {1:1}
9f5e38a0-b361-11e5-a211-45b7aa88107c system
sstable_activity 1451966757162 3134 730
{4:23}
621f05e0-b0ce-11e5-a211-45b7aa88107c system
sstable_activity 1451683616062 3060 719
{4:23}
5dde9ef0-b4b1-11e5-ac1a-45b7aa88107c system hints
1452110958175 9239287 9010439 {1:1}
8f94db70-b51b-11e5-bce5-45b7aa88107c system local
1452156568231 1201 688 {4:1}
4c414950-b32f-11e5-a211-45b7aa88107c system hints
1451945142885 14105741 13937247 {1:2}
bd0eb510-b388-11e5-a211-45b7aa88107c system hints
1451983557345 13062551 12823581 {1:2}
92707090-b09b-11e5-a211-45b7aa88107c system
sstable_activity 1451661792793 3149 689
{4:23}
aafc21e0-b497-11e5-ac1a-45b7aa88107c system hints
1452099920638 9542212 9382531 {1:1}
55e055e0-b263-11e5-a211-45b7aa88107c system hints
1451857541694 16600978 16364535 {1:1}
948bb3a0-b305-11e5-a211-45b7aa88107c system hints
1451927225306 14648744 14472833 {1:2}
7946c2a0-b162-11e5-a211-45b7aa88107c system hints
1451747220426 18970740 18970740 {1:1}
e5651290-b129-11e5-a211-45b7aa88107c system hints
1451722920505 18970740 18970740 {1:1}
b2226820-b05b-11e5-a211-45b7aa88107c system hints
1451634358178 18970740 18970740 {1:1}
11768640-b33f-11e5-a211-45b7aa88107c system hints
1451951916196 13937247 13717550 {1:2}
c05d07c0-b1fe-11e5-a211-45b7aa88107c system
sstable_activity 1451814341180 3329 852
{4:23}
029d1530-b376-11e5-a211-45b7aa88107c system hints
1451975513603 13204621 13062551 {1:2}
d1d10ac0-b441-11e5-ac1a-45b7aa88107c system
sstable_activity 1452063049068 3395 816
{1:3, 4:23}
0ec20100-b465-11e5-ac1a-45b7aa88107c system hints
1452078183696 10217777 10090370 {1:1}
b25689d0-b474-11e5-ac1a-45b7aa88107c system
sstable_activity 1452084900589 3221 776
{4:22}
50e84480-b321-11e5-a211-45b7aa88107c system hints
1451939137736 14276596 14105741 {1:2}
75ace380-b1e5-11e5-a211-45b7aa88107c system
sstable_activity 1451803478456 3171 820
{4:23}
d250c210-b441-11e5-ac1a-45b7aa88107c system
schema_columnfamilies 1452063049905 185693 46547
{1:1, 4:10}
As we can see, most compaction tasks come from keyspace system and
columnfamily hints. However hinted_handoff_enabled: false, so there may be
no new hints.
I checked the compaction settings of hints and sstable_activity.
hints:
compaction = {'enabled': 'false', 'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
sstable_activity:
compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
The background compaction is disabled for cf hints.
Most of the other use keyspaces are created by cassandra-cli. In previous
mail discussion, somebody says these cfs have memory and performance
issues. But as we can see most compactions come from keyspace system.
So what is source of compactions? Is high gc really related to the high
compactions?
--
*陈硕* *Shuo Chen*
[email protected]
[email protected]
