Yes, your request from the client is going to the LocalDC that you've defined for the data center aware load balancing policy, but with a consistency level of ONE, there is a chance for the coordinator (the node your client has connected to) to route the request across DC's.
Please see: https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dmlClientRequestsRead.html#dmlClientRequestsRead__two-dc-one "A two datacenter cluster with a consistency level of ONE "In a multiple datacenter cluster with a replication factor of 3, and a read consistency of ONE, the closest replica for the given row, regardless of datacenter, is contacted to fulfill the read request. In the background a read repair is potentially initiated, based on the read_repair_chance setting of the table, for the other replicas." A two datacenter cluster with a consistency level of LOCAL_ONE <https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dmlClientRequestsRead.html#dmlClientRequestsRead__two-dc-local-one> In a multiple datacenter cluster with a replication factor of 3, and a read consistency of LOCAL_ONE, the closest replica for the given row in the same datacenter as the coordinator node is contacted to fulfill the read request. In the background a read repair is potentially initiated, based on the read_repair_chance setting of the table, for the other replicas." Dynamic snitching also comes into play with reads. Just because your client is using TokenAware, and should connect to the appropriate replica node (which now is your coordinator) it can route your read request away from what it believes to be poorly performing nodes to another replica which could be in the other DC with CL = ONE. Read more about dynamic snitch here: https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchDynamic_c.html Regards, Eric Plowe On Wed, Mar 22, 2017 at 12:21 PM Shannon Carey <sca...@expedia.com> wrote: I understand all that, but it doesn't explain why the latency increases. The requests are not going to a remote DC. I know this because currently all requests are going to the client in one particular DC. The read request rate of the Cassandra nodes in the other DC remained flat (near zero) the whole time, compared to ~200read/s on the Cassandra nodes in the DC local to the client doing the reads. This is expected, because the DCAwareRoundRobinPolicy will cause local nodes to be used preferentially whenever possible. What's not expected is the dramatic latency increase. Btw this client is read-only: no writes. From: Eric Plowe <eric.pl...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Tuesday, March 21, 2017 at 7:23 PM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: ONE has much higher latency than LOCAL_ONE ONE means at least one replica node to ack the write, but doesn't require that the coordinator route the request to a node in the local data center. LOCAL_ONE was introduced to handle the case of when you have multiple data centers and cross data center traffic is not desirable. In multiple datacenter clusters, a consistency level of ONE is often desirable, but cross-DC traffic is not. LOCAL_ONEaccomplishes this. For security and quality reasons, you can use this consistency level in an offline datacenter to prevent automatic connection to online nodes in other datacenters if an offline node goes down. From: https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_config_consistency_c.html Regards, Eric On Tue, Mar 21, 2017 at 7:49 PM Shannon Carey <sca...@expedia.com> wrote: The cluster is in two DCs, and yes the client is deployed locally to each DC. From: Matija Gobec <matija0...@gmail.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Tuesday, March 21, 2017 at 2:56 PM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Re: ONE has much higher latency than LOCAL_ONE Are you running a multi DC cluster? If yes do you have application in both data centers/regions ? On Tue, Mar 21, 2017 at 8:07 PM, Shannon Carey <sca...@expedia.com> wrote: I am seeing unexpected behavior: consistency level ONE increases read latency 99th percentile to ~108ms (95th percentile to 5ms-90ms) up from ~5ms (99th percentile) when using LOCAL_ONE. I am using DSE 5.0 with Datastax client 3.0.0. The client is configured with a TokenAwarePolicy wrapping a DCAwareRoundRobinPolicy with usedHostsPerRemoteDc set to a very high number. Cassandra cluster has two datacenters. I would expect that when the cluster is operating normally (all local nodes reachable), ONE would behave the same as LOCAL_ONE. The Does anyone know why this is not the case?
