Hi As I understand that writes in cassandra are directly pushed to memory and using counters with CL.ONE shouldn't take the read latency for counters in account. So Writes for incrementing counters with CL.ONE should basically be really fast.
But in my 8 node cluster(16 core/32G ram/cassandra1.0.5/java7 each) with RF=2, At a traffic of 55k qps = 14k increments per node/7k write requests per node, the write latency(from jmx) increases to around 7-8 ms from the low traffic value of 0.5ms. The Nodes aren't even pushed with absent I/O, lots of free RAM and 30% CPU idle time/OS Load 20. The write latency by cfstats (supposedly the latency for 1 node to increment its counter) is a small amount (< 0.05ms). 1) Is the whole of 7-8ms being spent in thrift overheads and Scheduling delays ? (there is insignificant .1ms ping time between machines) 2) Do keeping a large number of CF(17 in our case) adversely affect write performance? (except from the extreme flushing scenario) 3) I see a lot of threads(4,000-10,000) with names like "pool-2-thread-*" (pointed out as client-connection-threads on the mailing list before) periodically forming up. but with idle cpu time and zero pending tasks in tpstats, why do requests keep piling up (GC stops threads for 100ms every 1-2 seconds, effectively pausing cassandra 5-10% of its time, but this doesn't seem to be the reason) Thanks Rohit
