The main guarantee LCS gives you is that most reads will only touch 1 row http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
If compaction is falling behind this may not hold. nodetool cfhistograms tells you how many SSTables were read from for reads. It's a recent histogram that resets each time you read from it. Also, parallel levelled compaction in 1.2 http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2 Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 20/01/2013, at 7:49 AM, Jim Cistaro <jcist...@netflix.com> wrote: > 1) In addition to iostat, dstat is a good tool to see wht kind of disck > throuput your are getting. That would be one thing to monitor. > 2) For LCS, we also see pending compactions skyrocket. During load, LCS will > create a lot of small sstables which will queue up for compaction. > 3) For us the biggest concern is not how high the pending count gets, but how > often it gets back down near zero. If your load is something you can do in > segments or pause, then you can see how fast the cluster recovers on the > compactions. > 4) One thing which we tune per cluster is the size of the files. Increasing > this from 5MB can sometimes improve things. But I forget if we have ever > changed this after starting data load. > > Is your cluster receiving read traffic during this data migration? If so, I > would say that read latency is your best measure. If the high number of > SSTables waiting to compact is not hurting your reads, then you are probably > ok. Since you are on SSD, there is a good chance the compactions are not > hurting you. As for compactionthroughput, we set ours high for SSD. You > usually wont use it all because the compactions are usually single threaded. > Dstat will help you measure this. > > I hope this helps, > jc > > From: Wei Zhu <wz1...@yahoo.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>, Wei Zhu > <wz1...@yahoo.com> > Date: Friday, January 18, 2013 12:10 PM > To: Cassandr usergroup <user@cassandra.apache.org> > Subject: Cassandra pending compaction tasks keeps increasing > > Hi, > When I run nodetool compactionstats > > I see the number of pending tasks keep going up steadily. > > I tried to increase the compactionthroughput, by using > > nodetool setcompactionthroughput > > I even tried the extreme to set it to 0 to disable the throttling. > > I checked iostats and we have SSD for data, the disk util is less than 5% > which means it's not I/O bound, CPU is also less than 10% > > We are using levelcompaction and in the process of migrating data. We have > 4500 writes per second and very few reads. We have about 70G data now and > will grow to 150G when the migration finishes. We only have one CF and right > now the number of SSTable is around 15000, write latency is still under > 0.1ms. > > Anything needs to be concerned? Or anything I can do to reduce the number of > pending compaction? > > Thanks. > -Wei > >
