Gabriel39 opened a new pull request, #69852:
URL: https://github.com/apache/doris/pull/69852
### What problem does this PR solve?
`vector_search` cannot currently select Lance's approximation policy. This
PR adds `"approx_mode"="fast|normal|accurate"`, validates the value before
metadata access, transports it as an optional typed Thrift field, and forwards
it to `lance_scanner_set_approx_mode` on each BE scanner. `EXPLAIN` shows an
explicitly requested mode.
Omitting the property preserves the existing native default (`normal` in the
pinned Lance revision). The setting applies through the shared single-vector
and multi-vector scanner setup. Existing refinement, probe counts, search
width, concurrency, split coverage, and distributed merging remain unchanged.
The existing lance-c dependency already provides this API, so no third-party
update or local patch is needed.
`docs/lance-approx-mode.md` documents mode-specific behavior, SQL usage,
interaction with other search options, rolling-upgrade compatibility, and the
local experiment below. In particular, `accurate` still selects approximate
candidates and does not guarantee exact Top-K or original-vector distances on a
quantized index. All participating BEs must support the new field to honor an
explicit mode consistently during a rolling upgrade.
### Local validation of behavior and effect
These are **native lance-c scanner measurements, not end-to-end Doris SQL
timings**. The library was freshly built from pinned lance-c
`a51ab47a9c814a746cb1714c99fad93dd3a188b8`, using Lance
`68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. The unoptimized development build
and Python/Arrow result conversion make the timings illustrative, not
production performance claims.
Procedure:
1. Generate 16,384 Gaussian Float32 vectors of dimension 128, then 128
independent query vectors, using xorshift64/Box-Muller with seed `20261029`.
Assign `row_id=0..16383` and `category=row_id%10`.
2. Build fully covering IVF_RQ (32 partitions, 5 bits), IVF_FLAT (32
partitions), and IVF_HNSW_FLAT (32 partitions, 16 edges, construction ef=100)
indexes. Select each index explicitly by its segment UUIDs.
3. Search squared L2 with k=10, nprobes=32, and query_parallelism=1. Repeat
IVF_RQ with refine_factor=4. For HNSW, set ef=64 and prefilter `category < 5`
(about 50% of rows).
4. Compute exact Top-10 independently with NumPy. Warm up each mode with
eight queries; run three rounds of 128 queries per mode, rotating mode order
per query and round. Each table cell below reports **mean Recall@10 / median
milliseconds**, from 384 searches. The document also includes p95.
5. Compare omitted mode with explicit normal over all 128 queries per case,
checking ordered IDs and Float32 distances. Independently recompute
original-vector distances for returned IVF_RQ rows to measure score error.
| Case | fast | normal | accurate |
| --- | ---: | ---: | ---: |
| IVF_RQ, no refinement | 26.80% / 14.57 | 91.72% / 15.90 | 91.72% / 20.52 |
| IVF_RQ, refine_factor=4 | 54.30% / 20.50 | 99.92% / 23.44 | 99.92% / 28.19
|
| IVF_FLAT | 100% / 25.71 | 100% / 25.78 | 100% / 25.71 |
| IVF_HNSW_FLAT, 50% prefilter | 100% / 40.45 | 100% / 44.51 | 100% / 44.48 |
All searches returned ten rows. Omitted mode matched explicit normal exactly
in every case. IVF_FLAT and filtered HNSW returned identical ordered IDs and
distances across modes. The data seed fixes vectors; ANN index training/layout
can vary across rebuilds.
Without refinement, normal and accurate had equal mean recall but different
ordered IDs on 89/128 queries. Mean absolute squared-L2 score error was 30.7325
for fast, 0.7986 for normal, and 0.7783 for accurate. Thus accurate slightly
improved distance estimation on this fixture without improving mean recall.
With refinement factor 4, normal and accurate returned identical results, while
fast still missed many neighbors. The mode is not interchangeable with
refinement, and its effect depends on the index/path and data.
### Release note
Add the optional `approx_mode` property to Lance `vector_search`, accepting
`fast`, `normal`, and `accurate`, with the existing default preserved when
omitted.
### Check List (For Author)
- Test
- [x] Unit tests: focused source compilation and JUnit execution of
`VectorSearchTableValuedFunctionTest` and `LanceScanNodeTest`: **55 tests
passed**. Coverage includes accepted/invalid modes, typed Thrift round trips,
default omission, and EXPLAIN.
- [x] Manual test: native mode comparison described above, plus
independent distance-error checks.
- [x] FE `mvn -B clean checkstyle:check` passed; clang-format 16 checks
passed for both affected C++ files; generated Thrift sources rebuilt
successfully.
- [x] Fresh BE reader and test source objects compiled successfully
against the new headers and pinned C API.
- [ ] Full BE unit-test execution: not run locally because the available
prebuilt BE libraries have an incompatible ABI. New request-validation coverage
is included for CI.
- [ ] SQL regression execution: added mode pass-through, EXPLAIN, and
invalid-value coverage to `test_lance_vector_search`; the full suite parsed
successfully with Groovy 4.0.19, but execution is pending CI.
- Behavior changed:
- [x] Yes. Explicit `approx_mode` now selects the native policy;
omission preserves existing behavior.
- Does this need documentation?
- [x] Yes. Included in `docs/lance-approx-mode.md`.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
