Hi, I have a question about what to expect when running a cluster across
datacenters with Local Quorum consistency.
My simplistic assumption is that the performance of an 8 node cluster split
across 2 data centers and running with local quorum would perform roughly
the same as a 4 node cluster in one data center.
I'm 95% certain we've set up the keyspace so that the entire range is in one
datacenter and the client is local. I see the keyspace split across all the
local nodes, with remote nodes owning 0%. Yet when I run the stress tests
against this configuration with local quorum, I see dramatically different
results from when I ran the same tests against a 4 node cluster. I'm still
5% unsure of this because the documentation on how to configure this is
pretty thin.
My understanding of Local Quorum was that once the data was written to a
local quorum, the commit would complete. I also believed that this would
eliminate any WAN latency required for replication to the other DC.
It not just that the split cluster runs slower, its also that there is
enormous variability in identical tests. Sometimes by a factor of 2 or more.
It seems as though the WAN latency is not only impacting performance, but
that it's also introducing a wide variation on overally performance.
Should WAN latency be completely hidden with local quorum? Or are there
second order issues involved that will impact performance??
I'm running in EC2 across us-east/west regions. I already know how
unpredictable EC2 performance can be, but what I'm seeing with here is far
beyond normal.performance variability for EC2
Is there something obvious that I'm missing that would explain why the
results are so different??
Here's the config when we run a 2x2 cluster:
Address DC Rack Status State Load
Owns Token
85070591730234615865843651857942052865
192.168.2.1 us-east 1b Up Normal 25.26 MB
50.00% 0
192.168.2.6 us-west 1c Up Normal 12.68 MB
0.00% 1
192.168.2.2 us-east 1b Up Normal 12.56 MB
50.00% 85070591730234615865843651857942052864
192.168.2.7 us-west 1c Up Normal 25.48 MB
0.00% 85070591730234615865843651857942052865
Thanks in advance.
CM
