Also, are you using incremental repairs (not sure about the available options in Spotify Reaper) what command did you run ?
2016-02-11 17:33 GMT+01:00 Alain RODRIGUEZ <arodr...@gmail.com>: > CPU load is fine, SSD disks below 30% utilization, no long GC pauses > > > > What is your current compaction throughput ? The current value of > 'concurrent_compactors' (cassandra.yaml or through JMX) ? > > nodetool getcompactionthroughput > > How to speed up compaction? Increased compaction throughput and concurrent >> compactors but no change. Seems there is plenty idle resources but can't >> force C* to use it. >> > > You might want to try un-throttle the compaction throughput through: > > nodetool setcompactionsthroughput 0 > > Choose a canari node. Monitor compaction pending and disk throughput (make > sure server is ok too - CPU...) > > Some other information could be useful: > > What is your number of cores per machine and the compaction strategies for > the 'most compacting' tables. What are write/update patterns, any TTL or > tombstones ? Do you use a high number of vnodes ? > > Also what is your repair routine and your values for gc_grace_seconds ? > When was your last repair and do you think your cluster is suffering of a > high entropy ? > > You can lower the stream throughput to make sure nodes can cope with what > repairs are feeding them. > > nodetool getstreamthroughput > nodetool setstreamthroughput X > > C*heers, > > ----------------- > Alain Rodriguez > France > > The Last Pickle > http://www.thelastpickle.com > > 2016-02-11 16:55 GMT+01:00 Michał Łowicki <mlowi...@gmail.com>: > >> Hi, >> >> Using 2.1.12 across 3 DCs. Each DC has 8 nodes. Trying to run repair >> using Cassandra Reaper but nodes after couple of hours are full of pending >> compaction tasks (regular not the ones about validation) >> >> CPU load is fine, SSD disks below 30% utilization, no long GC pauses. >> >> How to speed up compaction? Increased compaction throughput and >> concurrent compactors but no change. Seems there is plenty idle >> resources but can't force C* to use it. >> >> Any clue where there might be a bottleneck? >> >> >> -- >> BR, >> Michał Łowicki >> >> >
