Serhiy Bzhezytskyy created SOLR-18346:
-----------------------------------------
Summary: Leader-to-replica update retry depends on which exception
the client reports outermost
Key: SOLR-18346
URL: https://issues.apache.org/jira/browse/SOLR-18346
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Serhiy Bzhezytskyy
h2. Problem
{{SolrCmdDistributor}} decides whether a failed leader-to-replica update may be
retried by testing
one fixed position in the exception cause chain, so the decision depends on how
the client happened
to wrap the failure rather than on what actually failed.
{{StdNode.checkRetry}} tests the root cause when the outermost exception is a
{{SolrServerException}}, and the outermost exception otherwise:
{code:java}
if (err.e instanceof SolrServerException) {
if (isRetriableException(((SolrServerException) err.e).getRootCause())) {
return true;
}
} else {
if (isRetriableException(err.e)) {
return true;
}
}
return false;
{code}
{{ForwardNode.checkRetry}} tests exactly two positions for {{ConnectException}}:
{code:java}
if (err.e instanceof SolrServerException
&& ((SolrServerException) err.e).getRootCause() instanceof
ConnectException) {
doRetry = true;
} else if (err.e instanceof ConnectException) {
doRetry = true;
}
{code}
Neither shape covers a retriable cause that sits somewhere else in the chain.
Two real cases:
* the async client reports a connection failure wrapped in an
{{ExecutionException}}
* Jetty's {{ClientConnector}} wraps the underlying failure in a
{{SocketException}} of its own
In those cases the failure is not recognised as retriable, the update is not
retried, and the replica
is marked behind via {{ZkShardTerms.ensureTermsIsHigher}} and sent into
recovery — on a transient
network glitch that a retry would have absorbed.
h2. Proposed fix
Walk the cause chain instead of testing a fixed position, keeping the same set
of retriable types
({{SocketException}} / {{SocketTimeoutException}} for {{StdNode}},
{{ConnectException}} for
{{ForwardNode}}). The walk is depth-bounded, because a cause chain can be
cyclic — as the TODO on
{{SolrException.getRootCause}} already notes.
This widens nothing: no exception type becomes retriable that was not retriable
before. It only stops
the outcome depending on which wrapper happens to be outermost.
h2. Notes
* Split out of SOLR-7177 / PR #4638 at Mark Miller's suggestion there: that the
expectation of a
{{SolrServerException}} is fragile, and unrolling the cause chain to reach
the root cause is the
better shape.
* Scoped out of SOLR-9355, which was closed as Not A Problem: the behaviour
that issue described was
real when filed but the component it named ({{ConcurrentUpdateSolrClient}})
was never the gate, and
retry has always been decided in {{SolrCmdDistributor}}.
* {{ClosedChannelException}} retriability is deliberately *not* addressed here.
It is what the JDK
transport reports as the root cause of a dropped update connection and is
retriable nowhere in Solr
today; that is a separate question.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
