Those latencies look like the difference between a couple of disk seeks and reading something that's already in the os cache.
The dynamic snitch will favor nodes with lower latencies. Once a node has served enough reads, it might not have to hit disk very often, which produces lower latencies. So, if you have a hot dataset that fits into memory, the dynamic snitch starts a positive feedback loop where most reads will be served from one replica. I'm guessing the node with the low latencies is serving most of your reads. You can look at how quickly the total read count is increasing for each of the replicas to confirm this. It's not easy to do with only nodetool cfstats, but something like OpsCenter would help. On Fri, Jul 6, 2012 at 9:11 PM, Deno Vichas <d...@syncopated.net> wrote: > all, > > what would explain a huge different (12ms vs 0.1ms) in read latency from > node to node. i've got a 4 node cluster w/ replication factor of 3 using > hector. i'm seeing these numbers with nodetool cfstats. > > > thx, > deno > -- Tyler Hobbs DataStax <http://datastax.com/>
