FANNG1 opened a new issue, #66848: URL: https://github.com/apache/doris/issues/66848
## Description Doris currently executes one Lance `vector_search()` against the whole dataset with a single scan range. This preserves global Top-K semantics, but it cannot use multiple BEs to search a large Lance dataset in parallel. Add distributed execution for Lance vector search while preserving the current SQL semantics. The first implementation should be guarded by an experimental session variable (for example, `enable_distributed_lance_vector_search`) and keep the current single-scanner path as the default/fallback. ### Proposed execution model 1. Pin one Lance dataset version for the complete query lifecycle. 2. Plan indexed work by physical Lance vector-index segment. Keep each segment and the fragments it covers in one split. 3. Plan fragments not covered by the selected index as separate flat-search fallback splits. 4. For `use_index=false`, plan all relevant fragments as flat-search splits. 5. Each worker returns local candidates with `local_limit = top_k + offset` and `local_offset = 0`. 6. Doris exchanges and globally merges candidates ordered by `_distance`, then applies the global `offset` and returns `top_k` rows. Indexed and flat splits must be represented explicitly: - An indexed split carries index-segment UUIDs and forces indexed search. - A flat split carries fragment IDs and forces flat search. - A split must not mix index-segment UUIDs and fragment IDs. The planner may balance atomic segment/fragment work across available BEs. If the index metadata or coverage cannot be interpreted safely, it should fall back to the existing whole-dataset scanner instead of changing query results. ### Correctness requirements - Do not apply `offset` independently on each BE. A local offset can discard rows that belong to the global result. - Preserve partial-index behavior: appended/unindexed fragments must be searched unless the requested semantics explicitly allow them to be omitted. - Preserve `use_index=false` as distributed exact flat search. - Preserve the fixed dataset snapshot across metadata planning and execution. - Preserve the existing ordering between vector Top-K and outer SQL filters. - Keep `_distance` available as the internal global ordering key and as the result virtual column. ### Protocol and runtime work - Extend the FE/BE Lance scan-range protocol to carry index-segment UUIDs. - Use Lance/lance-c index-segment routing for indexed splits and fragment routing for flat splits. - Validate the two split forms in BE and reject ambiguous mixed input. - Reuse Doris local TopN + Exchange + merge TopN execution for the final global Top-K. - Add profile/EXPLAIN evidence that makes indexed versus flat execution diagnosable. ## Use case Large Lance datasets may contain many physical vector-index segments and newly appended fragments. Executing the complete nearest-neighbor search in one BE limits throughput and leaves the rest of a Doris cluster idle. Segment-aware fan-out allows Doris to search these independent units concurrently while returning the same globally ordered result shape as the current single-scanner implementation. ## Completion criteria - [ ] Multiple indexed segments are searched concurrently on multiple BEs. - [ ] Unindexed fragments participate through flat fallback. - [ ] `use_index=false` executes distributed flat search. - [ ] Global Top-K and non-zero `offset` are correct. - [ ] Planning and execution use the same pinned Lance dataset version. - [ ] Unsafe or unsupported metadata falls back to the current single-scanner path. - [ ] FE unit tests cover split planning, coverage, fallback, snapshot pinning, and global TopN construction. - [ ] BE unit tests cover indexed splits, flat splits, UUID validation, and invalid mixed splits. - [ ] A regression test with at least two BEs compares distributed and single-scanner results for indexed, partially indexed, flat, filtered, and offset queries. - [ ] Regression evidence proves actual ANN execution (for example, with a low-`nprobes` discriminator or an explicit runtime profile signal), rather than relying only on matching result rows. - [ ] User documentation describes enablement, semantics, fallback, and limitations. ## Related issues and implementations - Parent Lance capability tracking: #66340 - Additional Lance vector-index algorithm coverage: #66495 - Lance index lifecycle management: #66497 - Lance-Ray provides a reference segment-owned distributed vector-search execution model. ## Are you willing to submit PR? Yes, I am willing to submit a PR. ## Code of Conduct I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
