I have a 4 node cluster with lots JVM memory and lots of system memory that slows down when I'm doing lots of writes.
Running DataStax charts I see my read and write latency rise from 50-100 u-secs to 1500-4500 u-secs. This is across a 12 hour data load during which time the applied load is high but fairly constant (500-700 writes/sec). I'm trying to understand the slowdown: there is no memory pressure, I've run every option under nodetool to look for bottlenecks (tpstats, compactionStats, etc) and see none. I'm running with keycache and have about 98% hits. What can I check next? Thanks! Brian Tarbox
