Re: Manual compaction can't finish because of GC

2014-12-31 Thread Robert Coli
On Wed, Dec 31, 2014 at 12:44 PM, Mikhail Strebkov wrote: > I see, well that's what I expected, but it still should improve a read > latency, since it will reduce the number of disk seeks per row request, is > my assumption correct? > Yep. Also other things per-sstable, like bloom filters and in

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Mikhail Strebkov
I see, well that's what I expected, but it still should improve a read latency, since it will reduce the number of disk seeks per row request, is my assumption correct? On Wed, Dec 31, 2014 at 11:51 AM, Robert Coli wrote: > On Wed, Dec 31, 2014 at 11:35 AM, Mikhail Strebkov > wrote: > >> > How

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Robert Coli
On Wed, Dec 31, 2014 at 11:35 AM, Mikhail Strebkov wrote: > > How effective are compactions in this CF? "grep % > /var/log/cassandra/system.log"? > I'm not sure what type of logs should I share, when I do "grep % > /var/log/cassandra/system.log" I don't see anything related to this > long-running

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Mikhail Strebkov
> You can also set this online w/ nodetool, fyi. Oh, thanks! Will use this approach from now on > If you have SSD-esque devices, it is common to be CPU bound on compaction. We use spinning disks, unfortunately we can not afford SSDs in EC2 yet > How effective are compactions in this CF? "grep % /

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Robert Coli
On Wed, Dec 31, 2014 at 12:01 AM, Mikhail Strebkov wrote: > I set compaction_throughput_mb_per_sec to 0 and restarted Cassandra. > You can also set this online w/ nodetool, fyi. > It looks to me that compaction thread is busy doing something that > produces quite a lot of garbage for GC to coll

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Mikhail Strebkov
Hi Rob, Thanks for your response! Unthrottle compaction, that's an insane number of SSTables. I set compaction_throughput_mb_per_sec to 0 and restarted Cassandra. Somehow I don't see GC anymore but compaction is still very slow and IO is still not a bottleneck: iotop reports ~400 K/s for disk r

Re: Manual compaction can't finish because of GC

2014-12-30 Thread Robert Coli
On Tue, Dec 30, 2014 at 3:12 PM, Mikhail Strebkov wrote: > We have a table in our production Cassandra that is stored on 11369 > SSTables. The average SSTable count for the other tables is around 15, and > the read latency for them is much smaller. > Unthrottle compaction, that's an insane numbe