FANNG1 commented on issue #69573:
URL: https://github.com/apache/doris/issues/69573#issuecomment-6385587149
Thanks — the triage is right on both the mechanism and the correction, and I
have updated the description.
**Correction accepted.** My citation of
`test_lance_vector_search_legacy_index.groovy:84` was wrong: that
explain pins `version=1`, before the index was built, and asserts
`lanceVectorIndexStatus=NO_MATCH`. It is
a pure flat plan, not simultaneous mixed coverage.
**Sharper statement of the test gap**, after re-checking every suite that
asserts a non-zero
`lanceSearchUnindexedFragments`. Two suites do plan a mixed set of splits,
and neither can expose this:
- `test_lance_multivector_coverage.groovy:153` (`lanceSearchIndexSegments=2`,
`lanceSearchUnindexedFragments=1`) is multi-vector, so BE already forces
`refine_factor = 1`, and the
query passes `refine_factor=64` on top.
- `test_lance_search_snapshot.groovy`'s `qt_evolved_latest` *is* a
single-vector mixed-coverage query with
no `refine_factor` — `search_snapshot_evolved` version 5 has `vec_idx`
over fragment 0 and an appended,
uncovered fragment 1. But the fixture builds `IVF_FLAT`
(`lance_build_search_snapshot.py:130`), whose
distances are exact, and the query vector `[12.25,13.25,14.25,15.25]`
lands inside the *uncovered*
fragment, so the recorded top-3 (rows 12, 13, 11) all come from the flat
split. Nothing approximate ever
competes for a top-k slot.
So the missing shape is precise: single-vector, a **quantized** index
covering part of the dataset, at
least one uncovered fragment, no `refine_factor`, and a query vector near an
**indexed** row.
**On the proof standard** — agreed, and it changes what I will attach.
Comparing against `use_index=false`
is not evidence by itself, since `refine_factor=1` rescores the candidates
it already has and does not
widen selection. The check that isolates this defect is per-row: for each
returned row from an indexed
fragment, compare the reported `_distance` against the distance recomputed
from that row's stored vector,
and then check the merge order of indexed rows against uncovered rows. A
fixture with an analytic distance
function makes that a fixed expectation rather than a recall comparison —
the `legacy_vector_index`
fixture already does this (`vec = [row_id, row_id+1, row_id+2, row_id+3]`,
so squared L2 between rows `r`
and `n` is `4 * (n - r)^2`), which is a good base to copy: same vector
shape, `IVF_PQ` over the first
fragment only, then an append so the second fragment is uncovered.
Also noted, and I will keep both: the `IVF_FLAT` exemption, and the existing
distance-bound fallback,
which deliberately refuses quantized index scans with bounds because
candidates can be discarded before
rescoring. And no constant rescaling — quantization error is not one
multiplier.
I will build that fixture and post the runtime evidence you asked for
(generator, query vector, index
build options, pinned version, segment-to-fragment coverage, plus `EXPLAIN`
and `(row_id, _distance)` for
the default mixed query, the same query with `refine_factor=1`, and
`use_index=false`, labelled by which
fragment each row comes from) together with the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
