What compaction strategy? Looks like leveled – is that what you expect?
Any exceptions in the logs?
Are you throttling compaction?
SSD or spinning disks?
How many cores?
How many concurrent compactors?
From: Ezra Stuetzel <ezra.stuet...@riskiq.net>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, August 17, 2016 at 11:39 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: large number of pending compactions, sstables steadily increasing
I have one node in my cluster 2.2.7 (just upgraded from 2.2.6 hoping to fix
issue) which seems to be stuck in a weird state -- with a large number of
pending compactions and sstables. The node is compacting about 500gb/day,
number of pending compactions is going up at about 50/day. It is at about 2300
pending compactions now. I have tried increasing number of compaction threads
and the compaction throughput, which doesn't seem to help eliminate the many
pending compactions.
I have tried running 'nodetool cleanup' and 'nodetool compact'. The latter has
fixed the issue in the past, but most recently I was getting OOM errors,
probably due to the large number of sstables. I upgraded to 2.2.7 and am no
longer getting OOM errors, but also it does not resolve the issue. I do see
this message in the logs:
INFO [RMI TCP Connection(611)-10.9.2.218] 2016-08-17 01:50:01,985
CompactionManager.java:610 - Cannot perform a full major compaction as repaired
and unrepaired sstables cannot be compacted together. These two set of sstables
will be compacted separately.
Below are the 'nodetool tablestats' comparing a normal and the problematic
node. You can see problematic node has many many more sstables, and they are
all in level 1. What is the best way to fix this? Can I just delete those
sstables somehow then run a repair?
Normal node
keyspace: mykeyspace
Read Count: 0
Read Latency: NaN ms.
Write Count: 31905656
Write Latency: 0.051713177939359714 ms.
Pending Flushes: 0
Table: mytable
SSTable count: 1908
SSTables in each level: [11/4, 20/10, 213/100, 1356/1000, 306, 0, 0, 0,
0]
Space used (live): 301894591442
Space used (total): 301894591442
Problematic node
Keyspace: mykeyspace
Read Count: 0
Read Latency: NaN ms.
Write Count: 30520190
Write Latency: 0.05171286705620116 ms.
Pending Flushes: 0
Table: mytable
SSTable count: 14105
SSTables in each level: [13039/4, 21/10, 206/100, 831, 0, 0, 0, 0, 0]
Space used (live): 561143255289
Space used (total): 561143255289
Thanks,
Ezra
smime.p7s
Description: S/MIME cryptographic signature
