nick-boss-tech opened a new pull request, #5208: URL: https://github.com/apache/solr/pull/5208
🤖 *AI text below* 🤖 *(reviewed by Nick Shanin)* **A safe-leader test fails when the client sends an update to a replica the monkey stops.** `ChaosMonkeySafeLeaderTest` and `ChaosMonkeySafeLeaderWithPullReplicasTest` stop replicas but spare leaders, then assert that no update failed ([line 209](https://github.com/apache/solr/blob/b73f03976fa17b857bc0a81bc8af2c6b1c31e0bb/solr/core/src/test/org/apache/solr/cloud/ChaosMonkeySafeLeaderWithPullReplicasTest.java#L209)). The cloud client in these tests is built with a random choice between sending updates to shard leaders only and sending them to any replica ([line 370](https://github.com/apache/solr/blob/b73f03976fa17b857bc0a81bc8af2c6b1c31e0bb/solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java#L370)). In the second mode an update can go to a pull replica, and the monkey can stop that node while the request is in flight. The request then fails with `ClosedChannelException` and the check reports `expected:<0> but was:<1>`. On main with seed `E4EFD2B86 C0B81D4` on local disk, a `deleteById` sent to `collection1_shard1_replica_p11` failed 4 ms after the monkey stopped that node. **The safe-leader tests now use a leaders-only client.** Both tests override `createCloudClient` to build the client with `shardLeadersOnly` set. Leaders are the one kind of node these tests never stop, so an update that fails now is a real failure. Nothing else changes. The NothingIsSafe tests keep the random client. **Proof.** The failure is a timing race, so no single run fails on main. Forcing the any-replica client in `ChaosMonkeySafeLeaderWithPullReplicasTest` failed 1 of 4 random runs of `ChaosMonkeySafeLeaderWithPullReplicasTest` with `expected:<0> but was:<1>`. Its failed request was an update to `collection1_shard1_replica_p14`, a pull replica. With the leaders-only client ([compare](https://github.com/nick-boss-tech/solr/compare/b73f03976fa17b857bc0a81bc8af2c6b1c31e0bb...60eb0e7fd0521b8d866443a7b72dd623bacd884d), head 60eb0e7fd0), 8 runs of the two tests passed: `ChaosMonkeySafeLeaderWithPullReplicasTest` three times with the seed above and three times with random seeds, and `ChaosMonkeySafeLeaderTest` twice with random seeds. Nightly tests on, local disk, 2026-10-23. The runs were repeated on a second host and file system: 9 of 9 passed, including three runs of `ChaosMonkeySafeLeaderWithPullReplicasTest` at the seed above and three random-seed runs of each test. The full module chec k passes at the branch head. **The choice.** Three routes were available: 1. Use a leaders-only client in the safe-leader tests (this PR). 2. Leave failures caused by a stopped node out of the failure count. This also loosens the failure tolerance in the NothingIsSafe tests that read the same count. 3. Retry the cut-off update in the client. The load-balancing client does not retry an update after an I/O error, because the server may already have applied it ([line 576](https://github.com/apache/solr/blob/b73f03976fa17b857bc0a81bc8af2c6b1c31e0bb/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java#L576)). Context: the test name says leaders are safe, and a leader is the node every leaders-only update goes to. A retry on a non-idempotent update could apply it twice. Ladder: Does the test promise no failed update? Yes, but only for updates sent to a node it spares. Is a failed update to a node the monkey stops a defect? No. Does the any-replica client lose coverage? The NothingIsSafe tests still run it, with a failure tolerance. **Recommendation:** route 1. It keeps the zero-failure check strict, and it changes only the two tests that need it. **Limits.** The safe-leader tests no longer send updates through a non-leader replica under chaos. Whether the client should retry an update that fails with `ClosedChannelException` is a separate question; a follow-up ticket can be opened on request. The `Too many links` error that a network file system returned during an index copy ([IndexFetcher line 1224](https://github.com/apache/solr/blob/b73f03976fa17b857bc0a81bc8af2c6b1c31e0bb/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1224)) is not addressed. The timeout in `ChaosMonkeyNothingIsSafeWithPullReplicasTest` appeared only on that file system. *AI-generated description. Nick Shanin directed the work and takes responsibility for this contribution.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
