[
https://issues.apache.org/jira/browse/SOLR-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhenyuLi updated SOLR-18413:
----------------------------
Description:
h2. Description
During a SolrCloud {{MIGRATE}}, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key {{a!}} from {{source}} to {{target}} creates
state similar to:
{noformat}
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
{noformat}
The routing rule references the target collection by name. Deleting {{target}}
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
{noformat}
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
{noformat}
While this routing rule remains active, an update to {{source}} with a matching
ID such as {{a!2}} is first applied locally on the source shard leader.
The leader then enters {{DistributedZkUpdateProcessor.doDistribAdd()}}, which
resolves routing-rule destinations before distributing the update to the source
shard replicas:
{code:java}
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
{code}
For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
the target with:
{code:java}
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
{code}
If the target collection has already been deleted, this throws:
{noformat}
HTTP 400
Could not find collection : target
{noformat}
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
{noformat}
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 missing
client: HTTP 400
{noformat}
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent.
No update request is sent to the source replica. Therefore,
{{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
request does not lower the replica's shard term or directly trigger recovery.
Until another recovery mechanism repairs the replica:
* Queries may return different results depending on which replica serves the
request.
* A leader change may promote a replica that does not contain the update.
* The client cannot safely determine whether retrying the failed request is
appropriate.
* Retrying non-idempotent atomic updates could apply the operation more than
once.
h2. Execution flow
{noformat}
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X source replica distribution is never executed
{noformat}
h2. Reproduction
# Start a SolrCloud cluster with at least two nodes.
# Create a {{source}} collection with one shard and two replicas:
{{numShards=1}}, {{replicationFactor=2}}.
# Create a {{target}} collection: {{numShards=1}}, {{replicationFactor=1}}.
# Add and commit a document with a composite ID {{a!1}} to the {{source}}
collection.
# Run {{MIGRATE}} for route key {{a!}}:
{noformat}
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
{noformat}
# Confirm that {{source/shard1}} contains an unexpired routing rule for {{a!}}
whose target collection is {{target}}.
# Delete the {{target}} collection and wait until it is absent from
{{ClusterState}}.
# Before the routing rule expires, add another matching document ({{id = a!2}})
to the {{source}} collection. The request returns:
{noformat}
HTTP 400
Could not find collection : target
{noformat}
# Commit the {{source}} collection and query each source replica directly with
distributed querying disabled ({{q=id:"a!2"&distrib=false}}). The result is:
{noformat}
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 missing
{noformat}
h2. Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should:
# Stop forwarding updates through the stale routing rule.
# Remove the dangling rule on a best-effort basis.
# Continue distributing the update to all replicas of the source shard.
After the request completes:
{noformat}
client: update succeeds
source/shard1 leader: a!2 exists
source/shard1 replica: a!2 exists
source routing rule: dangling a! rule removed
{noformat}
was:
# Description
During a SolrCloud `MIGRATE`, the source slice receives a temporary routing
rule that forwards matching updates to the target collection.
For example, migrating route key `a!` from `source` to `target` creates state
similar to:
```text
source/shard1.routingRules["a!"] = {
targetCollection: "target",
...
}
```
The routing rule references the target collection by name. Deleting `target`
does not remove routing rules in other collections that still refer to it.
This can leave the cluster in the following state:
```text
source/shard1.routingRules["a!"].targetCollection = "target"
ClusterState:
source -> exists
target -> missing
```
While this routing rule remains active, an update to `source` with a matching
ID such as `a!2` is first applied locally on the source shard leader.
The leader then enters `DistributedZkUpdateProcessor.doDistribAdd()`, which
resolves routing-rule destinations before distributing the update to the source
shard replicas:
```java
final List<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState,
coll,
cmd.getIndexedIdStr(),
cmd.getSolrInputDocument());
if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
// Forward to the MIGRATE target.
...
}
if (nodes != null) {
// Distribute to replicas of the source shard.
cmdDistrib.distribAdd(...);
}
```
For a matching routing rule, `getNodesByRoutingRules()` currently resolves the
target with:
```java
DocCollection targetColl =
cstate.getCollection(rule.getTargetCollectionName());
```
If the target collection has already been deleted, this throws:
```text
HTTP 400
Could not find collection : target
```
At this point, however, the source leader has already applied the update
locally.
The exception prevents execution from reaching the subsequent source-replica
distribution block.
The resulting state can therefore be:
```text
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
client:
HTTP 400
```
This is more than a dangling-reference error. The request reports failure after
partially applying the mutation, leaving replicas of the source shard
inconsistent.
No update request is sent to the source replica. Therefore,
`SolrCmdDistributor` does not observe a replica failure. Consequently, this
request does not lower the replica's shard term or directly trigger recovery.
Until another recovery mechanism repairs the replica:
- Queries may return different results depending on which replica serves the
request.
- A leader change may promote a replica that does not contain the update.
- The client cannot safely determine whether retrying the failed request is
appropriate.
- Retrying non-idempotent atomic updates could apply the operation more than
once.
## Execution flow
```text
Client: ADD source / id=a!2
|
v
source shard leader
|
v
versionAdd()
|
v
local index/tlog update
|
v
getNodesByRoutingRules()
|
v
getCollection("target")
|
+---- target missing
|
v
HTTP 400 exception
|
X
source replica distribution
is never executed
```
# Reproduction
1. Start a SolrCloud cluster with at least two nodes.
2. Create a `source` collection with one shard and two replicas:
```text
collection = source
numShards = 1
replicationFactor = 2
```
3. Create a `target` collection:
```text
collection = target
numShards = 1
replicationFactor = 1
```
4. Add and commit a document with a composite ID to the `source` collection:
```text
id = a!1
```
5. Run `MIGRATE` for route key `a!`:
```text
source collection = source
target collection = target
split.key = a!
forward.timeout = 45
```
6. Confirm that `source/shard1` contains an unexpired routing rule for `a!`
whose target collection is `target`.
7. Delete the `target` collection and wait until it is absent from
`ClusterState`.
8. Before the routing rule expires, add another matching document to the
`source` collection:
```text
id = a!2
```
The request returns:
```text
HTTP 400
Could not find collection : target
```
9. Commit the `source` collection and query each source replica directly with
distributed querying disabled:
```text
q=id:"a!2"
distrib=false
```
The result is:
```text
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 missing
```
# Expected behavior
A routing rule whose target collection no longer exists should not abort a
source update after the source leader has already applied it.
The missing target should invalidate the dangling routing rule. Solr should:
1. Stop forwarding updates through the stale routing rule.
2. Remove the dangling rule on a best-effort basis.
3. Continue distributing the update to all replicas of the source shard.
After the request completes:
```text
client:
update succeeds
source/shard1 leader:
a!2 exists
source/shard1 replica:
a!2 exists
source routing rule:
dangling a! rule removed
```
> Deleting a MIGRATE target collection can leave source shard replicas
> inconsistent
> ---------------------------------------------------------------------------------
>
> Key: SOLR-18413
> URL: https://issues.apache.org/jira/browse/SOLR-18413
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.10.1
> Reporter: ZhenyuLi
> Priority: Major
>
> h2. Description
> During a SolrCloud {{MIGRATE}}, the source slice receives a temporary routing
> rule that forwards matching updates to the target collection.
> For example, migrating route key {{a!}} from {{source}} to {{target}} creates
> state similar to:
> {noformat}
> source/shard1.routingRules["a!"] = {
> targetCollection: "target",
> ...
> }
> {noformat}
> The routing rule references the target collection by name. Deleting
> {{target}} does not remove routing rules in other collections that still
> refer to it.
> This can leave the cluster in the following state:
> {noformat}
> source/shard1.routingRules["a!"].targetCollection = "target"
> ClusterState:
> source -> exists
> target -> missing
> {noformat}
> While this routing rule remains active, an update to {{source}} with a
> matching ID such as {{a!2}} is first applied locally on the source shard
> leader.
> The leader then enters {{DistributedZkUpdateProcessor.doDistribAdd()}}, which
> resolves routing-rule destinations before distributing the update to the
> source shard replicas:
> {code:java}
> final List<SolrCmdDistributor.Node> nodesByRoutingRules =
> getNodesByRoutingRules(
> clusterState,
> coll,
> cmd.getIndexedIdStr(),
> cmd.getSolrInputDocument());
> if (nodesByRoutingRules != null && !nodesByRoutingRules.isEmpty()) {
> // Forward to the MIGRATE target.
> ...
> }
> if (nodes != null) {
> // Distribute to replicas of the source shard.
> cmdDistrib.distribAdd(...);
> }
> {code}
> For a matching routing rule, {{getNodesByRoutingRules()}} currently resolves
> the target with:
> {code:java}
> DocCollection targetColl =
> cstate.getCollection(rule.getTargetCollectionName());
> {code}
> If the target collection has already been deleted, this throws:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> At this point, however, the source leader has already applied the update
> locally.
> The exception prevents execution from reaching the subsequent source-replica
> distribution block.
> The resulting state can therefore be:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> client: HTTP 400
> {noformat}
> This is more than a dangling-reference error. The request reports failure
> after partially applying the mutation, leaving replicas of the source shard
> inconsistent.
> No update request is sent to the source replica. Therefore,
> {{SolrCmdDistributor}} does not observe a replica failure. Consequently, this
> request does not lower the replica's shard term or directly trigger recovery.
> Until another recovery mechanism repairs the replica:
> * Queries may return different results depending on which replica serves the
> request.
> * A leader change may promote a replica that does not contain the update.
> * The client cannot safely determine whether retrying the failed request is
> appropriate.
> * Retrying non-idempotent atomic updates could apply the operation more than
> once.
> h2. Execution flow
> {noformat}
> Client: ADD source / id=a!2
> |
> v
> source shard leader
> |
> v
> versionAdd()
> |
> v
> local index/tlog update
> |
> v
> getNodesByRoutingRules()
> |
> v
> getCollection("target")
> |
> +---- target missing
> |
> v
> HTTP 400 exception
> |
> X source replica distribution is never executed
> {noformat}
> h2. Reproduction
> # Start a SolrCloud cluster with at least two nodes.
> # Create a {{source}} collection with one shard and two replicas:
> {{numShards=1}}, {{replicationFactor=2}}.
> # Create a {{target}} collection: {{numShards=1}}, {{replicationFactor=1}}.
> # Add and commit a document with a composite ID {{a!1}} to the {{source}}
> collection.
> # Run {{MIGRATE}} for route key {{a!}}:
> {noformat}
> source collection = source
> target collection = target
> split.key = a!
> forward.timeout = 45
> {noformat}
> # Confirm that {{source/shard1}} contains an unexpired routing rule for
> {{a!}} whose target collection is {{target}}.
> # Delete the {{target}} collection and wait until it is absent from
> {{ClusterState}}.
> # Before the routing rule expires, add another matching document ({{id =
> a!2}}) to the {{source}} collection. The request returns:
> {noformat}
> HTTP 400
> Could not find collection : target
> {noformat}
> # Commit the {{source}} collection and query each source replica directly
> with distributed querying disabled ({{q=id:"a!2"&distrib=false}}). The result
> is:
> {noformat}
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 missing
> {noformat}
> h2. Expected behavior
> A routing rule whose target collection no longer exists should not abort a
> source update after the source leader has already applied it.
> The missing target should invalidate the dangling routing rule. Solr should:
> # Stop forwarding updates through the stale routing rule.
> # Remove the dangling rule on a best-effort basis.
> # Continue distributing the update to all replicas of the source shard.
> After the request completes:
> {noformat}
> client: update succeeds
> source/shard1 leader: a!2 exists
> source/shard1 replica: a!2 exists
> source routing rule: dangling a! rule removed
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
