Can you slice up the "slows down" part a little more? Are you saying you are getting 4500 u-sec write latency ?
Are you using secondary indexes? What sort of read queries are slowing down? What does the schema look like ? If the simple checks like CPU, iostat and GC logging in the cassandra log do not show anything the next thing to look at is work load. Look for things like wide gets on wide rows, long lived rows with many deletes. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 8/01/2013, at 3:39 AM, Brian Tarbox <tar...@cabotresearch.com> wrote: > I have a 4 node cluster with lots JVM memory and lots of system memory that > slows down when I'm doing lots of writes. > > Running DataStax charts I see my read and write latency rise from 50-100 > u-secs to 1500-4500 u-secs. This is across a 12 hour data load during which > time the applied load is high but fairly constant (500-700 writes/sec). > > I'm trying to understand the slowdown: there is no memory pressure, I've run > every option under nodetool to look for bottlenecks (tpstats, > compactionStats, etc) and see none. I'm running with keycache and have about > 98% hits. > > What can I check next? > Thanks! > > Brian Tarbox
