FANNG1 opened a new pull request, #66512:
URL: https://github.com/apache/doris/pull/66512
### What problem does this PR solve?
Part of #66495 (tracked by #66340). Related: #65730.
Problem Summary:
The `vector_search()` query path is algorithm-agnostic — Lance selects the
index automatically — but until now only flat search was actually exercised:
the fixture from #65730 never created the vector index its tests claimed to
cover (the referenced `create_vector_search_index.py` was never committed), so
Lance silently fell back to flat KNN and the "IVF_PQ" regression proved nothing
about index usage.
This PR makes index usage real and provable for all six vector index
algorithms: **IVF_FLAT, IVF_PQ, IVF_SQ, IVF_HNSW_FLAT, IVF_HNSW_SQ,
IVF_HNSW_PQ**. No FE/BE/SQL/Thrift/lance-c changes — the deliverable is a
verified fixture plus regression coverage.
#### 1. Offline fixture generator
`docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py`,
pinned to pylance 4.0.1 (matching lance-c v0.1.2 → lance-rs 4.0.1 used by the
BE):
- Builds the Directory V2 catalog through the namespace API (never
predicting hashed storage paths): re-registers `all_types`, creates the `doris`
namespace and one indexed table per algorithm.
- Deterministic data: 1024 rows, two 512-row fragments, 16-dim Float32,
`embedding[j] = (row_id - 1) + j`. For a query equal to row r's vector, the
exact squared L2 distance of row n is `16 * (n - r)^2` — an integer ladder,
tie-free at the endpoints, exact in Float32.
- Self-check asserts: Lance plans contain `ANNSubIndex`/`ANNIvfPartition`;
every index covers both fragments; IVF_FLAT full-probe equals flat search
exactly; and the row-256 partition-boundary discriminator holds per table.
- The generated catalog (1.2M) is committed; the manifest is compacted back
to one version / one data file / three scalar indexes.
#### 2. Regression suites
- `test_lance_vector_search` now targets the real `vector_search_ivf_pq`
table; five new algorithm suites added.
- **Silent-fallback discriminator in every suite**: querying the row-256
partition boundary with `nprobes=1` must differ from the flat result. A
pipeline that ignores `use_index`/`nprobes` returns exactly the flat rows and
fails the assertion — golden files alone cannot catch this.
- IVF_FLAT additionally asserts `indexed == flat` programmatically at full
partition probes (its algorithm guarantee — vectors stored unquantized). Lossy
algorithms (PQ/SQ/HNSW_*) freeze `refine_factor`-reranked results in goldens
without claiming algorithm guarantees.
- HNSW suites assert the `ef` contract: `ef` below `top_k * refine_factor`
must raise `ef must be greater than or equal to k` instead of silently
truncating.
#### 3. Verified compatibility
The pylance-4.0.1-generated manifest was exercised against **lance-java
4.0.0** (the version inside lance-spark-bundle 0.4.0, which registers
`predicate_pushdown` at container startup) and **lance-java 9.1.0-beta.3** (the
FE Directory Namespace client): list/describe/register/deregister all work, and
the pylance self-check passes on the java-mutated manifest.
Recorded fixture behaviors (pinned Lance version): cosine query on an L2
index warns and falls back to flat search (no error); raw PQ distances are
quantization approximations, exact after `refine_factor` reranking.
#### Draft status
Draft until the `.out` goldens are generated by a full docker regression run
(`--forceGenOut`), which also provides the FE → BE → lance-c end-to-end
evidence. Everything above the Doris query layer is verified offline by the
generator self-check.
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [ ] No.
- [x] Yes. (compatibility matrix doc to follow in doris-website;
behaviors recorded by the generator)
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
https://claude.ai/code/session_014jKR13f5jdqs8NFyEuCnAN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
