Have a look at your column family histograms (nodetool cfhistograms iirc), if you notice things like a very long tail, a double hump or outliers it would indicate something wrong with your data model or you have a hot partition key/s.
Also looking at your 99 and 95 percentile latencies will just hide these occasional high latency reads as they fall outside these percentiles. If you are running a stock config, first rule out that its not your data model, then investigate things like disk latency, noisy neighbours (if you are on vms/ in the cloud). On 26 February 2015 at 03:01, Marcelo Valle (BLOOMBERG/ LONDON) < mvallemil...@bloomberg.net> wrote: > I am sorry if it's too basic and you already looked at that, but the first > thing I would ask would be the data model. > > What data model are you using (how is your data partitioned)? What queries > are you running? If you are using ALLOW FILTERING, for instance, it will be > very easy to say why it's slow. > > Most times people get slow queries in Cassandra they are using the wrong > data model. > > []s > > From: user@cassandra.apache.org > Subject: Re:Unexplained query slowness > > Our Cassandra database just rolled to live last night. I’m looking at our > query performance, and overall it is very good, but perhaps 1 in 10,000 > queries takes several hundred milliseconds (up to a full second). I’ve > grepped for GC in the system.log on all nodes, and there aren’t any recent > GC events. I’m executing ~500 queries per second, which produces negligible > load and CPU utilization. I have very minimal writes (one every few > minutes). The slow queries are across the board. There isn’t one particular > query that is slow. > > I’m running 2.0.12 with SSD’s. I’ve got a 10 node cluster with RF=3. > > I have no idea where to even begin to look. Any thoughts on where to start > would be greatly appreciated. > > Robert > > > -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr <http://twitter.com/instaclustr> | (650) 284 9692
