ZhenyuLi created SOLR-18413:
-------------------------------
Summary: Deleting a MIGRATE target collection can leave source
shard replicas inconsistent
Key: SOLR-18413
URL: https://issues.apache.org/jira/browse/SOLR-18413
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 9.10.1
Reporter: ZhenyuLi
# Description
During a SolrCloud MIGRATE, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key a! from source to target creates state
similar to:
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
The target collection is referenced by name. Deleting target does not remove
routing rules in other collections that refer to it. It is therefore possible
to have:
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
While this routing rule remains active, an add to source with an ID such as
a!2 is first applied locally on the source shard leader.
The leader subsequently enters DistributedZkUpdateProcessor.doDistribAdd(),
which resolves routing-rule destinations before distributing the update to the
source shard replicas:
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
For a matching routing rule, getNodesByRoutingRules() currently resolves the
target with:
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
If the target collection has been deleted, this throws:
HTTP 400
Could not find collection : target
At this point, however, the source leader has already applied the update
locally. The exception prevents execution from reaching the later
source-replica distribution block.
The resulting state can be:
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 is missing
client:
HTTP 400
This is more than a dangling-reference error. The request reports failure
after partially applying the mutation and leaves the source shard replicas
inconsistent.
No update request was sent to the source replica, so SolrCmdDistributor does
not observe a replica failure. Consequently, this request does not lower the
replica's shard term or trigger
recovery.
Until another recovery mechanism repairs the replica, queries can return
different results depending on which replica is selected. A leader change can
also promote a replica that does not
contain the update. The client cannot safely determine whether retrying the
failed request is appropriate; retries of non-idempotent atomic updates could
apply the operation more than once.
A simplified execution graph is:
Client: ADD source / id=a!2
|
v
source shard leader
versionAdd()
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+-- target missing
|
v
HTTP 400 exception
|
X
source replica distribution is not executed
# Reproduction
1. Start a SolrCloud cluster with at least two nodes.
2. Create a source collection with one shard and two replicas:
collection = source
numShards = 1
replicationFactor = 2
3. Create a target collection:
collection = target
numShards = 1
replicationFactor = 1
4. Add and commit a document with a composite ID to the source collection:
id = a!1
5. Run MIGRATE for route key a!:
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
6. Confirm that source/shard1 contains an unexpired routing rule for a! whose
target collection is target.
7. Delete the target collection and wait until it is absent from ClusterState.
8. Before the routing rule expires, add another matching document to the
source collection:
id = a!2
The request returns:
HTTP 400
Could not find collection : target
9. Commit the source collection and query each source replica directly with
distributed querying disabled:
q=id:"a!2"
distrib=false
The source shard leader contains a!2, while the other source replica does not.
# Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should
stop forwarding through that rule, remove it on a best-effort basis, and
continue distributing the update to all
replicas of the source shard.
After the request completes:
client:
update succeeds
source shard leader:
a!2 exists
source shard replica:
a!2 exists
source routing rule:
dangling a! rule is removed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
