I didn't know about this cfhistograms thing, very nice! From: user@cassandra.apache.org Subject: Re: Unexplained query slowness
Have a look at your column family histograms (nodetool cfhistograms iirc), if you notice things like a very long tail, a double hump or outliers it would indicate something wrong with your data model or you have a hot partition key/s. Also looking at your 99 and 95 percentile latencies will just hide these occasional high latency reads as they fall outside these percentiles. If you are running a stock config, first rule out that its not your data model, then investigate things like disk latency, noisy neighbours (if you are on vms/ in the cloud). On 26 February 2015 at 03:01, Marcelo Valle (BLOOMBERG/ LONDON) <mvallemil...@bloomberg.net> wrote: I am sorry if it's too basic and you already looked at that, but the first thing I would ask would be the data model. What data model are you using (how is your data partitioned)? What queries are you running? If you are using ALLOW FILTERING, for instance, it will be very easy to say why it's slow. Most times people get slow queries in Cassandra they are using the wrong data model. []s From: user@cassandra.apache.org Subject: Re:Unexplained query slowness Our Cassandra database just rolled to live last night. I’m looking at our query performance, and overall it is very good, but perhaps 1 in 10,000 queries takes several hundred milliseconds (up to a full second). I’ve grepped for GC in the system.log on all nodes, and there aren’t any recent GC events. I’m executing ~500 queries per second, which produces negligible load and CPU utilization. I have very minimal writes (one every few minutes). The slow queries are across the board. There isn’t one particular query that is slow. I’m running 2.0.12 with SSD’s. I’ve got a 10 node cluster with RF=3. I have no idea where to even begin to look. Any thoughts on where to start would be greatly appreciated. Robert -- Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | (650) 284 9692
