nick-boss-tech opened a new pull request, #5135: URL: https://github.com/apache/solr/pull/5135
🤖 *AI text below* 🤖 *(posted on behalf of Nick Shanin)* https://issues.apache.org/jira/browse/SOLR-6975 ## What happens today **A distributed search that sorts on `_docid_` fails while the coordinating node merges the shard responses.** With `q=*:*`, `fl=id,test_sS`, `sort=_docid_ asc` and `rows=20` on a multi shard collection, the request throws `IndexOutOfBoundsException: Index: 0, Size: 0`. The exception comes from `NamedList.getName` in the shard comparator, under `QueryComponent.mergeIds` ([ticket with the stack trace](https://issues.apache.org/jira/browse/SOLR-6975)). The comparator file has not changed on main since this branch was cut, so the path is still there. ## What this change does **A `_docid_` sort no longer reads a sort value from the shards. Results from different shards are grouped by shard.** Shards do not send `_docid_` values back, so the merge has nothing to compare for that key. The comparator for this key now reports a tie ([ShardFieldSortedHitQueue.java L123-L126](https://github.com/nick-boss-tech/solr/blob/761aa629bb83e241df2aefc114513797b7339522/solr/core/src/java/org/apache/solr/handler/component/ShardFieldSortedHitQueue.java#L123-L126)). Documents from one shard keep the order that shard returned, in both directions. Documents from different shards are grouped by shard address, and the sort direction does not change that grouping ([lessThan, L82-L111](https://github.com/nick-boss-tech/solr/blob/761aa629bb83e241df2aefc114513797b7339522/solr/core/src/java/org/apache/solr/handler/component/ShardFieldSortedHitQueue.java#L82-L111)). If `_docid_` is one of several sort keys, the merge ignores that key. The other keys decide the order, and the shard address breaks what is left. ## Proof **The new test fails on the base commit and passes with this change.** - [`DistributedQueryComponentOptimizationTest`](https://github.com/nick-boss-tech/solr/blob/761aa629bb83e241df2aefc114513797b7339522/solr/core/src/test/org/apache/solr/handler/component/DistributedQueryComponentOptimizationTest.java#L348-L362) 10 of 10 passed at `761aa629bb83`. - On the base commit `e432df19c4a`, with only the new test added, `testSortByDocId` fails with the same `IndexOutOfBoundsException` from `NamedList.getName` under the shard comparator, during `QueryComponent.mergeIds`. - The test sends an ascending and a descending `_docid_` request. It checks that the number of returned documents equals `numFound` and is above zero. It does not check the order across shards, because that order is not defined. ## A choice to check **Documents from different shards come back grouped by shard. The order across shards is not defined.** Options considered: 1. Group by shard, as implemented. The request succeeds. Each shard keeps its own order, and the groups follow the shard address. 2. Reject a distributed `_docid_` sort with a clear error. The cost: users of multi shard collections get an error where they get results today. Was grouping by shard the right call, or should distributed `_docid_` sorts fail with a clear error? ## Limits **Multi key `_docid_` sorts are not tested, and a separate payload fetch failure is not fixed here.** - The new test covers one `_docid_` key. The behavior with several keys is described above but not tested. - A payload fetch failure in a distributed query also happens on the base commit. I did not look into it here, and this change does not touch it. I can open a follow-up ticket and PR for it on request. Changelog: [changelog/unreleased/SOLR-6975-distributed-docid-sort.yml](https://github.com/nick-boss-tech/solr/blob/761aa629bb83e241df2aefc114513797b7339522/changelog/unreleased/SOLR-6975-distributed-docid-sort.yml) ### AI assistance AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
