Isaac Reath created CASSANDRA-21654:
---------------------------------------
Summary: Coordinator node experiences read failures on ungraceful
shutdown of replica
Key: CASSANDRA-21654
URL: https://issues.apache.org/jira/browse/CASSANDRA-21654
Project: Apache Cassandra
Issue Type: Bug
Components: Legacy/Coordination
Reporter: Isaac Reath
Assignee: Isaac Reath
When a replica node dies, the coordinator node can keep routing reads to it
until the failure detector convicts the node. During this time, messages will
continue to collect on the {{OutboundConnection}} of the dead replica until it
reaches capacity, at which point the {{RequestCallbacks}} calls
{{ReadCallback#onFailure}} which aborts the entire request, despite there being
sufficient replicas alive to service the request.
For RF=3 and CL=QUORUM, a single replica failure causes the whole query to
abort unless the table is configured with {{speculative_retry = 'ALWAYS'}}. For
higher replication factors (e.g., rf=5, rf=9), this prevents a read from
succeeding even with {{speculative_retry = 'ALWAYS'}} and multiple concurrent
node failures.
This ticket proposes adding a second trigger for speculative retry beyond
latency. In the event of an overloaded connection, we propose speculating
immediately to another replica, letting the query succeed so long as there are
sufficient replicas to satisfy consistency.
This issue was originally hit on a production cluster with 18 nodes running
4.1.10 with roughly 2k reads/sec/node. I've also been able to reproduce using
an in-JVM dtest off of cassandra-4.1 HEAD.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
