> > CPU load is fine, SSD disks below 30% utilization, no long GC pauses
What is your current compaction throughput ? The current value of 'concurrent_compactors' (cassandra.yaml or through JMX) ? nodetool getcompactionthroughput How to speed up compaction? Increased compaction throughput and concurrent > compactors but no change. Seems there is plenty idle resources but can't > force C* to use it. > You might want to try un-throttle the compaction throughput through: nodetool setcompactionsthroughput 0 Choose a canari node. Monitor compaction pending and disk throughput (make sure server is ok too - CPU...) Some other information could be useful: What is your number of cores per machine and the compaction strategies for the 'most compacting' tables. What are write/update patterns, any TTL or tombstones ? Do you use a high number of vnodes ? Also what is your repair routine and your values for gc_grace_seconds ? When was your last repair and do you think your cluster is suffering of a high entropy ? You can lower the stream throughput to make sure nodes can cope with what repairs are feeding them. nodetool getstreamthroughput nodetool setstreamthroughput X C*heers, ----------------- Alain Rodriguez France The Last Pickle http://www.thelastpickle.com 2016-02-11 16:55 GMT+01:00 Michał Łowicki <mlowi...@gmail.com>: > Hi, > > Using 2.1.12 across 3 DCs. Each DC has 8 nodes. Trying to run repair using > Cassandra Reaper but nodes after couple of hours are full of pending > compaction tasks (regular not the ones about validation) > > CPU load is fine, SSD disks below 30% utilization, no long GC pauses. > > How to speed up compaction? Increased compaction throughput and concurrent > compactors but no change. Seems there is plenty idle resources but can't > force C* to use it. > > Any clue where there might be a bottleneck? > > > -- > BR, > Michał Łowicki > >
