I don't know what exactly compaction logs is like in system.log. But I see
logs like this in system.log, I think maybe this is the compaction log
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,664 StatusLogger.java (line
118) ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.local 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.peers 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.batchlog 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.compactions_in_progress 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.compaction_history 9,2650
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.paxos 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,670 StatusLogger.java (line
121) system.schema_columns 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.range_xfers 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.peer_events 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.hints 0,0
INFO [ScheduledTasks:1] 2016-01-09 22:51:02,671 StatusLogger.java (line
121) system.sstable_activity 69,20930
...
To clarify the source of compaction, I restarted the cluster and truncate
system.hints at Jan 8. However, it also shows compactionhistory of hints
like this. I checked the timestamp however, many history items are compared
at Jan 6.
-b313-11e5-a211-45b7aa88107c system hints
1451933296208 14472833 14276596 {1:2}
7a7e5ad0-b42c-11e5-a211-45b7aa88107c system
sstable_activity 1452053883133 3215 852
{4:23}
16e97320-b264-11e5-a211-45b7aa88107c system
sstable_activity 1451857865554 3114 726
{4:23}
f8ab7d40-b62d-11e5-aa30-45b7aa88107c system
sstable_activity 1452274426644 3047 751
{4:23}
ea955a00-b173-11e5-a211-45b7aa88107c system hints
1451754711968 18970740 18970740 {1:1}
cf98c3c0-b54e-11e5-bce5-45b7aa88107c system
sstable_activity 1452178579964 3503 964
{4:23}
9e74bf80-b4c0-11e5-ac1a-45b7aa88107c system
sstable_activity 1452117508984 3135 800
{4:22}
4b9ffb20-b27d-11e5-a211-45b7aa88107c system
sstable_activity 1451868691410 3037 791
{4:23}
4f059400-b3e0-11e5-a211-45b7aa88107c system
sstable_activity 1452021168448 2985 802
{1:2, 4:23}
9de3b670-b439-11e5-ac1a-45b7aa88107c system hints
1452059525975 10532201 10550449 {2:2}
f1c2f710-b439-11e5-ac1a-45b7aa88107c system hints
1452059666689 10479415 10479415 {1:1}
9b4f7700-b439-11e5-ac1a-45b7aa88107c system hints
1452059521648 10594437 10532000 {1:1,
2:1}
402f42d0-b2fc-11e5-a211-45b7aa88107c system
sstable_activity 1451923218301 3184 776
{4:23}
47589f00-b231-11e5-a211-45b7aa88107c system
sstable_activity 1451836042480 3212 785
{4:23}
848ee140-b439-11e5-ac1a-45b7aa88107c system hints
1452059483476 10648367 10601270 {1:1,
2:1}
8736f390-b4f9-11e5-ac1a-45b7aa88107c system hints
1452141951305 8261575 8112577 {1:1}
ff591b80-b37a-11e5-a211-45b7aa88107c system
sstable_activity 1451977655608 3025 765
{4:23}
Besides, there are also lots pending compaction tasks over 70000000.
On Sat, Jan 9, 2016 at 2:42 AM, Robert Coli <[email protected]> wrote:
>
>
> On Thu, Jan 7, 2016 at 6:30 PM, Shuo Chen <[email protected]> wrote:,
>>
>> I am using Cassandra 2.0.16 with 4 nodes and found too many compactions
>> for this cluster. This caused too much full gc and choked the system. I
>> have discussed the high gc in previous mails but didnot get the satisfied
>> answers.
>>
>> To clarify the source of the compactions, I shutdown all the clients and
>> there is no reading and writing requests outside. Besides,
>> hinted_handoff_enabled: false is set. The cluster is restarted yesterday
>> and have run for 1.5 days.
>>
>
> Do you see compactions of the hints CF in the system.log?
>
> =Rob
>
>
--
*陈硕* *Shuo Chen*
[email protected]
[email protected]
