I have hit dead-ends every where I turned on this issue. We had a 15-node cluster that was doing 35 ms all along for years. At some point, we made a decision to shrink it to 13. Read latency rose to near 70 ms. Shortly after, we decided this was not acceptable, so we added the three nodes back in. Read latency dropped to near 50 ms and it has been hovering around this value for over 6 months now.
Repairs run regularly, load on cluster nodes is even, application activity profile has not changed. Why are we unable to get back the same read latency now that the cluster is 15 nodes large same as it was before? -- ---------------------------------------- Thank you