FANNG1 commented on issue #69573: URL: https://github.com/apache/doris/issues/69573#issuecomment-6397336186
Lance-level measurement of the mechanism, plus one correction to the description above. **Correction: the ordering claim should be conditional.** The description says "the ranking is distorted". What is unconditional is that the `_distance` column mixes two scales; whether the final order also changes depends on whether the quantization error exceeds the gap to the neighbouring exactly-scored row, which is data- and codebook-dependent. Measured below. **Fixture.** `partial_vector_index.lance`, built with the writer pinned in `lance_fixture_requirements.txt` (pylance 7.0.0): row_id 1..512 in fragment 0 with `vec_idx` (IVF_PQ, l2, 2 partitions, 2 sub-vectors, 8 bits, `index_file_version="V3"`) over it, then row_id 513..640 appended as fragment 1, which no index covers. `vec = [row_id, row_id + 1, row_id + 2, row_id + 3]` as float32, so the squared L2 distance between rows `r` and `n` is exactly `4 * (n - r)^2`. Query `[512.25, 513.25, 514.25, 515.25]`, `nprobes=2` (= the partition count, so the indexed arm probes every partition), `k=6`. The two Doris arms were reproduced at the Lance level: the indexed arm with `fast_search=True`, which restricts the scan to the data the index covers, and the uncovered fragment's rows, whose distances are exact by definition. pylance 7 cannot run a fragment-scoped `nearest` (`Not supported: This operation is not supported for fragment scan`), which is precisely the capability lance-c gives Doris and the reason Doris can split this way at all. **Indexed arm, no refinement** (what Doris plans today) vs **`refine_factor=1`**: | row_id | fragment | arm | reported, no refinement | reported, refine_factor=1 | analytic | |---|---|---|---|---|---| | 512 | 0 | indexed | **1.2808** | 0.25 | 0.25 | | 513 | 1 | flat | 2.25 | 2.25 | 2.25 | | 511 | 0 | indexed | **4.3113** | 6.25 | 6.25 | | 514 | 1 | flat | 12.25 | 12.25 | 12.25 | | 510 | 0 | indexed | **25.25** | 20.25 | 20.25 | | 515 | 1 | flat | 30.25 | 30.25 | 30.25 | So the indexed rows are reported 1.03 too high, 1.94 too low and 5.0 too high, while the rows from the uncovered fragment in the same result are exact. `refine_factor=1` reproduces the analytic values exactly, and an exact scan of the same rows agrees with it. **Why the order survived here.** The gaps between adjacent true distances near the top are 2, 4, 6 and 10, and the errors above are smaller than the gap each row would have to cross, so the merged order stayed `[512, 513, 511, 514, 510, 515]`. The error is a property of the trained codebook, not of the data shape: a rebuild with identical parameters reported 0.53, 3.53 and 20.25 for the same three rows. An order inversion therefore cannot be made deterministic by fixture design, so the regression pins the rescored side — the analytic constants — which is stable, rather than pinning a particular wrong order. Two parameter notes for anyone reproducing this: `num_bits=4` over this ladder is not a usable fixture, because it collapses the index so the true neighbours are not selected at all, which is a recall failure rather than the scale mismatch under test; and `num_sub_vectors=1` with 8 bits happens to reproduce the exact distances, so it does not exhibit the defect either. **Implementation status.** The fix is written against `branch-4.1` along the lines agreed above: a new `TLanceFileDesc` field (id 7) set only on the index-segment splits of a mixed plan whose selected index is not `IVF_FLAT` / `IVF_HNSW_FLAT`, carried to BE beside the existing `use_scalar_index` flag, applied in `_configure_vector_search` before the user options block so a larger explicit `refine_factor` still wins, with a guard rejecting the flag on a split that carries no index segment, and `EXPLAIN` reporting `lanceSearchRefineDistances`. FE compiles and `LanceScanNodeTest` passes with the new planner cases (40 tests). The BE unit tests and the end-to-end regression are not run yet; the Doris-side `EXPLAIN` and `(row_id, _distance)` outputs will come with the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
