[
https://issues.apache.org/jira/browse/SOLR-18346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105114#comment-18105114
]
ASF subversion and git services commented on SOLR-18346:
--------------------------------------------------------
Commit ed893b5faaf4a89ed1dbdb8efa7990b0675a9d95 in solr's branch
refs/heads/main from Serhiy Bzhezytskyy
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=ed893b5faaf ]
SOLR-18346: decide update retries from the cause chain, not one fixed position
in it (#4678)
A transient connection failure from a shard leader to one of its replicas is
now retried when the failure arrives wrapped inside another exception, instead
of sending the replica into recovery. Previously whether the retry happened
depended on which exception the client reported outermost.
> Leader-to-replica update retry depends on which exception the client reports
> outermost
> --------------------------------------------------------------------------------------
>
> Key: SOLR-18346
> URL: https://issues.apache.org/jira/browse/SOLR-18346
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Reporter: Serhiy Bzhezytskyy
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h2. Problem
> {{SolrCmdDistributor}} decides whether a failed leader-to-replica update may
> be retried by testing
> one fixed position in the exception cause chain, so the decision depends on
> how the client happened
> to wrap the failure rather than on what actually failed.
> {{StdNode.checkRetry}} tests the root cause when the outermost exception is a
> {{SolrServerException}}, and the outermost exception otherwise:
> {code:java}
> if (err.e instanceof SolrServerException) {
> if (isRetriableException(((SolrServerException) err.e).getRootCause())) {
> return true;
> }
> } else {
> if (isRetriableException(err.e)) {
> return true;
> }
> }
> return false;
> {code}
> {{ForwardNode.checkRetry}} tests exactly two positions for
> {{ConnectException}}:
> {code:java}
> if (err.e instanceof SolrServerException
> && ((SolrServerException) err.e).getRootCause() instanceof
> ConnectException) {
> doRetry = true;
> } else if (err.e instanceof ConnectException) {
> doRetry = true;
> }
> {code}
> Neither shape covers a retriable cause that sits somewhere else in the chain.
> Two real cases:
> * the async client reports a connection failure wrapped in an
> {{ExecutionException}}
> * Jetty's {{ClientConnector}} wraps the underlying failure in a
> {{SocketException}} of its own
> In those cases the failure is not recognised as retriable, the update is not
> retried, and the replica
> is marked behind via {{ZkShardTerms.ensureTermsIsHigher}} and sent into
> recovery — on a transient
> network glitch that a retry would have absorbed.
> h2. Proposed fix
> Walk the cause chain instead of testing a fixed position, keeping the same
> set of retriable types
> ({{SocketException}} / {{SocketTimeoutException}} for {{StdNode}},
> {{ConnectException}} for
> {{ForwardNode}}). The walk is depth-bounded, because a cause chain can be
> cyclic — as the TODO on
> {{SolrException.getRootCause}} already notes.
> This widens nothing: no exception type becomes retriable that was not
> retriable before. It only stops
> the outcome depending on which wrapper happens to be outermost.
> h2. Notes
> * Split out of SOLR-7177 / PR #4638 at Mark Miller's suggestion there: that
> the expectation of a
> {{SolrServerException}} is fragile, and unrolling the cause chain to reach
> the root cause is the
> better shape.
> * Scoped out of SOLR-9355, which was closed as Not A Problem: the behaviour
> that issue described was
> real when filed but the component it named ({{ConcurrentUpdateSolrClient}})
> was never the gate, and
> retry has always been decided in {{SolrCmdDistributor}}.
> * {{ClosedChannelException}} retriability is deliberately *not* addressed
> here. It is what the JDK
> transport reports as the root cause of a dropped update connection and is
> retriable nowhere in Solr
> today; that is a separate question.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
