FANNG1 opened a new pull request, #66779:
URL: https://github.com/apache/doris/pull/66779
### What problem does this PR solve?
Issue Number: Part of #66495
Problem Summary: test_lance_vector_search documents its doris.vector_search
fixture as carrying an IVF_PQ index, but the fixture SQL
(run07_create_vector_types.sql) delegated index creation to a companion
create_vector_search_index.py that was never committed, and
lance-spark-bundle
0.4.0 cannot create vector indexes through SQL. The table therefore had no
index at all, so every use_index / nprobes / refine_factor query in the suite
silently executed a flat KNN scan while the goldens still looked correct.
Nothing in the repository could observe the difference.
Reproduction: build the table the old fixture built and probe it with
nprobes=1. Lance ignores nprobes on an unindexed dataset and returns exactly
the flat top-10 (rows 256,255,257,254,258,253,259,252,260,251 for the
boundary
query) - identical to the flat baseline, which is why the defect was
invisible.
Fix: replace the Spark-created table with an offline-generated Directory
Namespace V2 catalog that carries a real IVF_PQ index, and add the evidence
that the index is actually used.
- lance_build_preinstalled_catalog.py rebuilds the committed fixture and
self-checks it: exactly one IVF_PQ index named embedding_ivf_pq_f32
covering
every fragment, ANNSubIndex and ANNIvfPartition present in the indexed
plan,
KNNVectorDistance and no ANN node in the flat plan, and the exact
16 * (n - r)^2 distance ladder that every golden and comment encodes, so a
change to the data shape fails here instead of surfacing as an opaque
golden
diff. Index creation goes through the physical dataset because
DirectoryNamespace.create_table_index raises UnsupportedOperationError.
- doris.vs_ivf_pq_f32 replaces doris.vector_search: 1024 rows in two
fragments, 16-dimensional Float32 embedding[j] = (row_id - 1) + j, so the
exact squared L2 distance between rows r and n is 16 * (n - r)^2 and the
head/tail queries have no distance ties. Columns are declared NOT NULL to
match the fixture being replaced, keeping the only non-nullable Lance
column
mapping recorded by any Lance suite's DESC golden. The
vs_<algorithm>_<element type>
name encodes one cell of the algorithm x element type matrix, so a missing
combination is visible from the table list alone.
- The suite gains a silent-fallback discriminator. Row 256 sits on the first
IVF partition boundary, so a genuine single-partition probe must miss true
neighbours from the next partition. The suite asserts that the nprobes=1
distance sequence differs from flat search; on the previous unindexed
fixture the two are identical and the assertion fails. Distances are
compared rather than row ids because the boundary query is symmetric and
rows r-d and r+d tie. top_k is 9 there, the last cut that lands on a
complete tie pair: at 10 the pair at distance 400 is split, so the golden
would pin an arbitrary winner that any change to Lance's top-k selection
could flip. Which partition edge row 256 lands next to changes on every
retrain, so no measured range is hardcoded; --check prints it instead.
- IVF_PQ is lossy, so every indexed query uses refine_factor and the suite
documents indexed/flat agreement as an observed property of this frozen
fixture and pinned Lance version, not an algorithm guarantee.
The fixture is generated with the pins in lance_fixture_requirements.txt.
pylance 4.0.1 matches the Lance line the BE uses through lance-c v0.1.2
(verified: the built binary embeds lance-c-0.1.2 / lance-core-4.0.1), and the
generated catalog stays readable by lance-java 4.0.0, which registers
doris.predicate_pushdown into the same __manifest at container startup.
Index training is not bit-reproducible, so regenerating the fixture changes
the binary output; the reproducible properties are asserted by the generator
self-check instead. IVF_FLAT, IVF_SQ, IVF_HNSW_* and the other vector element
types are follow-up work for #66495.
### Release note
None
### Check List (For Author)
- Test: Regression test
- Fixture generator self-check with the pinned dependencies
- test_lance_vector_search regenerated with -forceGenOut, then passed the
normal golden comparison
- The whole external_table_p0/lance directory passed (6 suites, 0
failed),
covering the pre-existing suites that share the regenerated __manifest
- Cross-checked that the nprobes=1 golden row order matches what pylance
records probing the same physical index directly
- Behavior changed: No, test fixture and regression coverage only
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
