[
https://issues.apache.org/jira/browse/SOLR-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated SOLR-18413:
----------------------------
Description:
# Description
During a SolrCloud `MIGRATE`, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key `a!` from `source` to `target` creates state
similar to:
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
The routing rule references the target collection by name. Deleting `target`
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
While this routing rule remains active, an update to `source` with a matching
ID such as `a!2` is first applied locally on the source shard leader.
The leader then enters `DistributedZkUpdateProcessor.doDistribAdd()`, which
resolves routing-rule destinations before distributing the update to the source
shard replicas:
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
For a matching routing rule, `getNodesByRoutingRules()` currently resolves the
target with:
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
If the target collection has already been deleted, this throws:
HTTP 400
Could not find collection : target
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
client:
HTTP 400
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent.
No update request is sent to the source replica. Therefore,
`SolrCmdDistributor` does not observe a replica failure, and this request does
not lower the replica's shard term or directly trigger recovery.
Until another recovery mechanism repairs the replica:
- Queries may return different results depending on which replica serves the
request.
- A leader change may promote a replica that does not contain the update.
- The client cannot safely determine whether the failed request should be
retried.
- Retrying non-idempotent atomic updates could apply the operation more than
once.
## Execution flow
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X
source replica distribution
is never executed
# Reproduction
1. Start a SolrCloud cluster with at least two nodes.
2. Create a `source` collection with one shard and two replicas:
collection = source
numShards = 1
replicationFactor = 2
3. Create a `target` collection:
collection = target
numShards = 1
replicationFactor = 1
4. Add and commit a document with a composite ID to `source`:
id = a!1
5. Run `MIGRATE` for route key `a!`:
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
6. Confirm that `source/shard1` contains an unexpired routing rule for `a!`
whose target collection is `target`.
7. Delete the `target` collection and wait until it is absent from
`ClusterState`.
8. Before the routing rule expires, add another matching document to `source`:
id = a!2
The request returns:
HTTP 400
Could not find collection : target
9. Commit the `source` collection and query each source replica directly with
distributed querying disabled:
q=id:"a!2"
distrib=false
The result is:
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
# Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied that update locally.
The missing target should invalidate the dangling routing rule.
Solr should:
1. Stop forwarding updates through the stale routing rule.
2. Remove the dangling rule on a best-effort basis.
3. Continue distributing the update to all replicas of the source shard.
After the request completes, the expected state is:
client:
update succeeds
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 exists
source routing rule:
dangling a! rule removed
was:
# Description
During a SolrCloud MIGRATE, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key a! from source to target creates state
similar to:
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
The target collection is referenced by name. Deleting target does not remove
routing rules in other collections that refer to it. It is therefore possible
to have:
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
While this routing rule remains active, an add to source with an ID such as
a!2 is first applied locally on the source shard leader.
The leader subsequently enters DistributedZkUpdateProcessor.doDistribAdd(),
which resolves routing-rule destinations before distributing the update to the
source shard replicas:
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
For a matching routing rule, getNodesByRoutingRules() currently resolves the
target with:
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
If the target collection has been deleted, this throws:
HTTP 400
Could not find collection : target
At this point, however, the source leader has already applied the update
locally. The exception prevents execution from reaching the later
source-replica distribution block.
The resulting state can be:
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 is missing
client:
HTTP 400
This is more than a dangling-reference error. The request reports failure
after partially applying the mutation and leaves the source shard replicas
inconsistent.
No update request was sent to the source replica, so SolrCmdDistributor does
not observe a replica failure. Consequently, this request does not lower the
replica's shard term or trigger
recovery.
Until another recovery mechanism repairs the replica, queries can return
different results depending on which replica is selected. A leader change can
also promote a replica that does not
contain the update. The client cannot safely determine whether retrying the
failed request is appropriate; retries of non-idempotent atomic updates could
apply the operation more than once.
A simplified execution graph is:
Client: ADD source / id=a!2
|
v
source shard leader
versionAdd()
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+-- target missing
|
v
HTTP 400 exception
|
X
source replica distribution is not executed
# Reproduction
1. Start a SolrCloud cluster with at least two nodes.
2. Create a source collection with one shard and two replicas:
collection = source
numShards = 1
replicationFactor = 2
3. Create a target collection:
collection = target
numShards = 1
replicationFactor = 1
4. Add and commit a document with a composite ID to the source collection:
id = a!1
5. Run MIGRATE for route key a!:
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
6. Confirm that source/shard1 contains an unexpired routing rule for a! whose
target collection is target.
7. Delete the target collection and wait until it is absent from ClusterState.
8. Before the routing rule expires, add another matching document to the
source collection:
id = a!2
The request returns:
HTTP 400
Could not find collection : target
9. Commit the source collection and query each source replica directly with
distributed querying disabled:
q=id:"a!2"
distrib=false
The source shard leader contains a!2, while the other source replica does not.
# Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should
stop forwarding through that rule, remove it on a best-effort basis, and
continue distributing the update to all
replicas of the source shard.
After the request completes:
client:
update succeeds
source shard leader:
a!2 exists
source shard replica:
a!2 exists
source routing rule:
dangling a! rule is removed
> Deleting a MIGRATE target collection can leave source shard replicas
> inconsistent
> ---------------------------------------------------------------------------------
>
> Key: SOLR-18413
> URL: https://issues.apache.org/jira/browse/SOLR-18413
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
>
> # Description
> During a SolrCloud `MIGRATE`, the source slice receives a temporary routing
> rule that forwards matching updates to the target collection.
> For example, migrating route key `a!` from `source` to `target` creates state
> similar to:
> source/shard1.routingRules["a!"] = {
> targetCollection: "target",
> ...
> }
> The routing rule references the target collection by name. Deleting `target`
> does not remove routing rules in other collections that still refer to it.
> This can leave the cluster in the following state:
> source/shard1.routingRules["a!"].targetCollection = "target"
> ClusterState:
> source -> exists
> target -> missing
> While this routing rule remains active, an update to `source` with a matching
> ID such as `a!2` is first applied locally on the source shard leader.
> The leader then enters `DistributedZkUpdateProcessor.doDistribAdd()`, which
> resolves routing-rule destinations before distributing the update to the
> source shard replicas:
> final List<SolrCmdDistributor.Node> nodesByRoutingRules =
> getNodesByRoutingRules(
> clusterState,
> coll,
> cmd.getIndexedIdStr(),
> cmd.getSolrInputDocument());
> if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
> // Forward to the MIGRATE target.
> ...
> }
> if (nodes != null) {
> // Distribute to replicas of the source shard.
> cmdDistrib.distribAdd(...);
> }
> For a matching routing rule, `getNodesByRoutingRules()` currently resolves
> the target with:
> DocCollection targetColl =
> cstate.getCollection(rule.getTargetCollectionName());
> If the target collection has already been deleted, this throws:
> HTTP 400
> Could not find collection : target
> At this point, however, the source leader has already applied the update
> locally.
> The exception prevents execution from reaching the subsequent source-replica
> distribution block.
> The resulting state can therefore be:
> source/shard1 leader:
> a!2 exists
> source/shard1 replica:
> a!2 missing
> client:
> HTTP 400
> This is more than a dangling-reference error. The request reports failure
> after partially applying the mutation, leaving replicas of the source shard
> inconsistent.
> No update request is sent to the source replica. Therefore,
> `SolrCmdDistributor` does not observe a replica failure, and this request
> does not lower the replica's shard term or directly trigger recovery.
> Until another recovery mechanism repairs the replica:
> - Queries may return different results depending on which replica serves the
> request.
> - A leader change may promote a replica that does not contain the update.
> - The client cannot safely determine whether the failed request should be
> retried.
> - Retrying non-idempotent atomic updates could apply the operation more than
> once.
> ## Execution flow
> Client: ADD source / id=a!2
> |
> v
> source shard leader
> |
> v
> versionAdd()
> |
> v
> local index/tlog update
> |
> v
> getNodesByRoutingRules()
> |
> v
> getCollection("target")
> |
> +---- target missing
> |
> v
> HTTP 400 exception
> |
> X
> source replica distribution
> is never executed
> # Reproduction
> 1. Start a SolrCloud cluster with at least two nodes.
> 2. Create a `source` collection with one shard and two replicas:
> collection = source
> numShards = 1
> replicationFactor = 2
> 3. Create a `target` collection:
> collection = target
> numShards = 1
> replicationFactor = 1
> 4. Add and commit a document with a composite ID to `source`:
> id = a!1
> 5. Run `MIGRATE` for route key `a!`:
> source collection = source
> target collection = target
> split.key = a!
> forward.timeout = 45
> 6. Confirm that `source/shard1` contains an unexpired routing rule for `a!`
> whose target collection is `target`.
> 7. Delete the `target` collection and wait until it is absent from
> `ClusterState`.
> 8. Before the routing rule expires, add another matching document to `source`:
> id = a!2
> The request returns:
> HTTP 400
> Could not find collection : target
> 9. Commit the `source` collection and query each source replica directly with
> distributed querying disabled:
> q=id:"a!2"
> distrib=false
> The result is:
> source/shard1 leader:
> a!2 exists
> source/shard1 replica:
> a!2 missing
> # Expected behavior
> A routing rule whose target collection no longer exists should not abort a
> source update after the source leader has already applied that update locally.
> The missing target should invalidate the dangling routing rule.
> Solr should:
> 1. Stop forwarding updates through the stale routing rule.
> 2. Remove the dangling rule on a best-effort basis.
> 3. Continue distributing the update to all replicas of the source shard.
> After the request completes, the expected state is:
> client:
> update succeeds
> source/shard1 leader:
> a!2 exists
> source/shard1 replica:
> a!2 exists
> source routing rule:
> dangling a! rule removed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
