nick-boss-tech opened a new pull request, #5098: URL: https://github.com/apache/solr/pull/5098
🤖 *AI text below* 🤖 *(posted on behalf of Nick Shanin)* https://issues.apache.org/jira/browse/SOLR-18530 ## What happens today **An add made after the test cuts off one replica can fail, and the test stops there.** `RecoveryAfterSoftCommitTest` closes one replica's socket proxy, then adds more documents through the normal cloud client. That client can send an add to the cut replica. The add fails with `SolrServerException: IOException occurred when talking to server at ...`, and its cause is a `ClosedChannelException`. The test fails before its recovery check. ## What this change does **After the cut, this test sends its adds only to the shard leader.** The adds before the cut still use the normal client. The adds after it use a new client that sends updates only to shard leaders ([test](https://github.com/nick-boss-tech/solr/blob/98e5368d996518b9f4a94f85d7c2fcd933d3f485/solr/core/src/test/org/apache/solr/cloud/RecoveryAfterSoftCommitTest.java#L102-L117)). The leader still forwards each add to the cut replica. A failed forward is logged and does not fail the add ([DistributedZkUpdateProcessor](https://github.com/nick-boss-tech/solr/blob/98e5368d996518b9f4a94f85d7c2fcd933d3f485/solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java#L1217-L1240)). The document count is unchanged, so the replica still misses more updates than its update log keeps. Our reading of the cause, which these runs do not prove: the cloud client still held a pooled connection to the cut replica's proxy. SolrJ moves an update to another endpoint only when it could not connect or the request was never sent ([LBSolrClient](https://github.com/nick-boss-tech/solr/blob/98e5368d996518b9f4a94f85d7c2fcd933d3f485/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBSolrClient.java#L675-L680)). A write that reached the network and then failed is not retried. The change touches only this test. No product code changes. ## Proof **This is a test-only change. The repeated runs pass, but they do not show that the change fixes the failure.** - `RecoveryAfterSoftCommitTest` with this change: 6 of 6 runs passed. The runs used the CI seed `A57A22E346C237C6` and five random seeds. Checked 2026-10-10 at 98e5368d996518b9f4a94f85d7c2fcd933d3f485. - The original test: 5 of 5 runs passed at the CI seed `A57A22E346C237C6`. It did not fail, so there is no before-and-after result. - The failure is intermittent and did not appear in these runs. This change is not shown to fix it. The reasoning rests on the code path described above. ## A choice to check **The adds after the cut use a leaders-only client, not a retry in the test.** The other option was to keep the normal client and retry the same add when it fails with `ClosedChannelException`. That would keep any-replica writes during the partition in this test, which the leaders-only client removes. The existing test `CheckRetryUnrollTest` already asserts that `ClosedChannelException` is not retried ([test](https://github.com/nick-boss-tech/solr/blob/98e5368d996518b9f4a94f85d7c2fcd933d3f485/solr/core/src/test/org/apache/solr/update/CheckRetryUnrollTest.java#L97-L103)), and changing that rule would be a separate decision. Was routing these adds through a leaders-only client the right call, or would you rather keep random routing and retry in the test? ## Limits - The failure did not reproduce in the runs above, so the fix is not shown to remove it. - Other tests that cut a replica's proxy and then write through the cloud client may have the same pattern. I have not checked them. A follow-up for those is available on request. - The test checks only that replicas become active within 30 seconds. It does not check which recovery path ran. The document count is what rules out peer sync. - The test helper also randomizes one routing setting for this client. The leader is still tried first. - SolrJ still does not retry a `ClosedChannelException` on an update. That is a separate product decision. ### AI assistance AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
