Re: Many SSTables only on one node

2018-04-09 Thread kurt greaves
If there were no other messages about anti-compaction similar to: > > SSTable YYY (ranges) will be anticompacted on range [range] Then no anti-compaction needed to occur and yes, it was not the cause. On 5 April 2018 at 13:52, Dmitry Simonov wrote: > Hi, Evelyn! > > I've found the following me

Re: Many SSTables only on one node

2018-04-05 Thread Dmitry Simonov
Hi, Evelyn! I've found the following messages: INFO RepairRunnable.java Starting repair command #41, repairing keyspace XXX with repair options (parallelism: parallel, primary range: false, incremental: false, job threads: 1, ColumnFamilies: [YYY], dataCenters: [], hosts: [], # of ranges: 768) IN

Re: Many SSTables only on one node

2018-04-05 Thread Evelyn Smith
It might not be what cause it here. But check your logs for anti-compactions. > On 5 Apr 2018, at 8:35 pm, Dmitry Simonov wrote: > > Thank you! > I'll check this out. > > 2018-04-05 15:00 GMT+05:00 Alexander Dejanovski >: > 40 pending compactions is pretty high a

Re: Many SSTables only on one node

2018-04-05 Thread Dmitry Simonov
Thank you! I'll check this out. 2018-04-05 15:00 GMT+05:00 Alexander Dejanovski : > 40 pending compactions is pretty high and you should have way less than > that most of the time, otherwise it means that compaction is not keeping up > with your write rate. > > If you indeed have SSDs for data st

Re: Many SSTables only on one node

2018-04-05 Thread Alexander Dejanovski
40 pending compactions is pretty high and you should have way less than that most of the time, otherwise it means that compaction is not keeping up with your write rate. If you indeed have SSDs for data storage, increase your compaction throughput to 100 or 200 (depending on how the CPUs handle th

Re: Many SSTables only on one node

2018-04-05 Thread Dmitry Simonov
Hi, Alexander! SizeTieredCompactionStrategy is used for all CFs in problematic keyspace. Current compaction throughput is 16 MB/s (default value). We always have about 40 pending and 2 active "CompactionExecutor" tasks in "tpstats". Mostly because of another (bigger) keyspace in this cluster. But

Re: Many SSTables only on one node

2018-04-05 Thread Alexander Dejanovski
Hi Dmitry, could you tell us which compaction strategy that table is currently using ? Also, what is the compaction max throughput and is auto-compaction correctly enabled on that node ? Did you recently run repair ? Thanks, On Thu, Apr 5, 2018 at 10:53 AM Dmitry Simonov wrote: > Hello! > > C

Many SSTables only on one node

2018-04-05 Thread Dmitry Simonov
Hello! Could you please give some ideas on the following problem? We have a cluster with 3 nodes, running Cassandra 2.2.11. We've recently discovered high CPU usage on one cluster node, after some investigation we found that number of sstables for one CF on it is very big: 5800 sstables, on othe