Gabriel39 opened a new pull request, #68613: URL: https://github.com/apache/doris/pull/68613
### What problem does this PR solve? Related PR: https://github.com/lance-format/lance/pull/9599 Correctness prerequisite: https://github.com/lance-format/lance/pull/9537 Unfiltered Lance ANN searches scoped to complete index segments can build an explicit allowlist containing every row ID in those segments. This adds work proportional to the segment row count even though index coverage already proves that the candidates belong to the scan scope. Carry the upstream segment-coverage optimization as a local third-party patch, including its 14 parameterized regression cases. Predicate filters, partial segment coverage and unknown coverage retain the prefilter. Existing deletion, visibility and unindexed-fragment handling remain in force. The existing pinned Lance revision also has a 4-bit PQ scoring discrepancy: omitting a prefilter can select bulk uint8 scoring instead of per-row float scoring, changing candidate ordering. Include the correctness portion of the upstream fix using the existing float loop and pairwise accumulation, with focused boundary/score/top-1 regressions. The newer upstream SIMD screening implementation is not imported. This can increase CPU cost for 4-bit PQ bulk scoring; end-to-end performance has not been benchmarked. Download the exact existing Lance revision as a checksum-verified companion of `lance_c`, apply the patches automatically, and switch the C API manifest and lockfile to relative source paths. Package versions and the C API stay unchanged. Separate source markers support fresh builds, previously patched caches and re-extraction without modifying shared Cargo caches. ### Release note Avoid redundant row-ID prefilter construction for unfiltered Lance vector searches over complete index segments. Preserve consistent 4-bit PQ candidate scoring with and without prefilters. ### Check List (For Author) - Test - [x] Unit Test: Lance scanner and PQ regression tests (results below). - [x] Manual test: third-party patch integration and locked dependency checks. - [ ] Doris SQL regression / complete BE build: not run locally; CI requested. - Behavior changed: - [x] Yes. Eligible ANN scans omit redundant allowlists; 4-bit PQ bulk scores match per-row scores. - Does this need documentation? - [x] No. No new SQL, configuration or index format. ### Validation - Patched Lance PQ suite: **104 passed**, including 85 new score/boundary cases. The new cases failed against the original kernel, including a deterministic top-1 ordering reversal; they pass with the prerequisite patch. - Patched Lance prefilter suite: **98 passed** (includes all 14 upstream cases). - C API: `cargo check --locked --tests` passed using the patched companion. `cargo metadata --locked` verified all 21 Lance workspace packages resolve from the companion, with no original Lance git source left in the lockfile. - Downloader regression: fresh extraction, idempotent rerun, companion re-extraction, existing C API cache upgrade and patch rejection passed. Existing checksum and fallback downloader tests also passed. - `cargo fmt --all --check`, shell syntax checks and `git diff --check` passed. Rust validation used toolchain 1.97.0. Tests ran against the exact pinned source after applying the Doris patches. The prefilter suite runs serially because a pre-existing randomized HNSW test can be flaky in parallel. A full Doris build, Doris SQL regression and an end-to-end benchmark remain for follow-up validation; this PR does not claim a measured latency gain. The build integration regression can be reproduced after downloading the archives: ```bash bash thirdparty/download-thirdparty.sh lance_c bash thirdparty/test/lance-prefilter-patch-test.sh thirdparty/src ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
