[
https://issues.apache.org/jira/browse/CASSANDRA-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820849#comment-16820849
]
Benedict commented on CASSANDRA-14983:
--------------------------------------
bq. If one node has a slow disk, more sstables or similar issues, should it act
mostly as a proxy to other nodes or do we accept slow requests?
Unfortunately we have no reliable mechanism of predicting that the present node
is slower than other nodes, besides the (not very good) dynamic snitch, which
should do what you suggest already and pick other nodes if we are slow.
I think it depends how we view our features. While we cannot guarantee
performing speculative retry under GC (and in this scenario may not even
respond to requests at all, so is a wider problem we need to progressively
address as a project), we can make a best effort to implement it as a feature
we advertise and define explicitly. There's a strong argument to be made that
we should not be knowingly introducing extra scenario where it doesn't invoke
when it should.
But, speculative retry is itself a guessing game, and one we don't play very
well (since we assume uniform request duration), and one that wastes cluster
resources when we get it wrong. Is it worth trading the possibility of a very
unbalanced node serving as coordinator and failing to perform speculative
retry, in exchange for capping the downside to incorrectly guessed speculative
retries? Probably, I think.
But really, in my view, it suggests we should be revisiting the logic behind
speculative retry altogether, incorporating the knowledge that request duration
is non-uniform, and knowingly using the local query execution time to inform
how long we should expect remote queries to take. But perhaps it is enough for
now to simply accept the tradeoff for this optimisation. It's worth noting,
though, that the optimisation is only really valuable for very short queries.
bq. I also want to mention that if we do want to have a feature that waits for
local requests before doing speculative retries I think we should keep it
separate from the fast path. The requests can and will go through the normal
path still and I guess we want to have a similar behavior when that happens.
Could you expand on what you mean here?
> Local reads potentially blocking remote reads
> ---------------------------------------------
>
> Key: CASSANDRA-14983
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14983
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
> Reporter: Marcus Olsson
> Assignee: Marcus Olsson
> Priority: Low
> Attachments: graph_local_read.html, graph_local_read_trunk.html,
> local_read_trace.log
>
>
> Since CASSANDRA-4718 there is a fast path allowing local requests to continue
> to [work in the same
> thread|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java#L157]
> rather than being sent over to the read stage.
> Based on the comment
> {code:java}
> // We delay the local (potentially blocking) read till the end to avoid
> stalling remote requests.
> {code}
> it seems like this should be performed last in the chain to avoid blocking
> remote requests but that does not seem to be the case when the local request
> is a data request. The digest request(s) are sent after the data requests are
> sent (and now the transient replica requests as well). When the fast path is
> used for local data/transient data requests this will block the next type of
> request from being sent away until the local read is finished and add
> additional latency to the request.
> In addition to this it seems like local requests are *always* data requests
> (might not be a problem), but the log message can say either ["digest" or
> "data"|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/reads/AbstractReadExecutor.java#L156]
> as the type of request.
> I have tried to run performance measurements to see the impact of this in 3.0
> (by moving local requests to the end of ARE#executeAsync()) but I haven't
> seen any big difference yet. I'll continue to run some more tests to see if I
> can find a use case affected by this.
> Attaching a trace (3.0) where this happens. Reproduction:
> # Create a three node CCM cluster
> # Provision data with stress (rf=3)
> # In parallel:
> ## Start stress read run
> ## Run multiple manual read queries in cqlsh with tracing on and
> local_quorum (as this does not always happen)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
