Gabriel39 opened a new pull request, #69339:
URL: https://github.com/apache/doris/pull/69339
### What problem does this PR solve?
Related PR: #69332 (lance-c dependency upgrade)
Ordinary SQL against Lance external tables does not use the existing Lance
search path for vector Top-N, and `score()` cannot produce Lance BM25
relevance. Users currently need explicit `vector_search()` /
`full_text_search()` TVFs.
This PR adds automatic rewrites on branch-4.1, reusing the existing
fixed-snapshot search planner, index-segment execution, global candidate merge,
and deferred payload reads. It upgrades lance-c to
`4fafcd64525b274b113208390bbcca86bbbc093b`, as in #69332, and directly uses
`lance_scanner_set_row_addr_allowlist()` introduced by lance-c #96.
### SQL examples and behavior changes
The examples assume a Lance table with a three-dimensional Float32 embedding
and a supported full-text index on `body`.
#### 1. Approximate vector Top-N
```sql
SELECT doc_id,
l2_distance_approximate(embedding, [0.1, 0.2, 0.3]) AS distance
FROM lance_catalog.docs.documents
WHERE tenant_id = 7
ORDER BY distance ASC
LIMIT 10 OFFSET 5;
```
Before: ordinary Lance scanning evaluates the scalar distance and Doris
performs Top-N; this SQL does not request native Lance vector search.
After: the optimizer creates a native Lance L2 search with the original
scalar predicate applied before candidate selection. Each search split requests
15 candidates with offset 0; Doris merges them and applies the final global
`OFFSET 5 LIMIT 10`. Compatible Lance vector indexes can serve the approximate
search through the existing search planner. Candidate selection may therefore
be approximate rather than an exhaustive scalar ranking.
The SQL result remains Euclidean distance: Lance's squared L2 result is
converted with `CAST(sqrt(_distance) AS FLOAT)`.
Inner-product queries use the corresponding descending order:
```sql
SELECT doc_id,
inner_product_approximate(embedding, [0.1, 0.2, 0.3]) AS similarity
FROM lance_catalog.docs.documents
WHERE tenant_id = 7
ORDER BY similarity DESC
LIMIT 10;
```
This becomes native Lance dot search. The projection converts Lance dot
distance with `1.0f - _distance`, so the SQL output remains an inner-product
similarity and higher is better.
#### 2. BM25 full-text Top-N
```sql
SELECT doc_id, score() AS relevance
FROM lance_catalog.docs.documents
WHERE body MATCH_ANY 'database search'
AND tenant_id = 7
ORDER BY relevance DESC
LIMIT 10;
```
Before: `score()` on this Lance external scan is unsupported.
After: one top-level MATCH predicate becomes a native prepared Lance
full-text query, `score()` reads its generated BM25 score, and Doris merges the
highest-scoring candidates. The native prepared query preserves the
dataset-wide scoring context across index segments and row-ID filtering.
- `MATCH_ANY` maps to a match query with OR.
- `MATCH_ALL` maps to a match query with AND.
- `MATCH_PHRASE` maps to a phrase query using the stored analyzer and
position information.
- A lower score bound such as `AND score() >= 0.5` is supported as a
residual over the generated score. Upper bounds and other score predicates are
rejected because filtering a pruned Top-K with those predicates could lose
valid results.
#### 3. Scalar WHERE to native row-ID prefilter
For the scalar predicates in the examples, the execution path is:
```text
Bound Lance snapshot + validated Substrait predicate
-> row-ID-only native scan (no user-column projection)
-> native UINT64 _rowid allowlist
-> vector / prepared full-text search
-> Doris global Top-N and final projection
```
The candidate scanner and search scanner use the same pinned dataset
snapshot. A query-local cache shares the candidate list across search readers
using the dataset URI, version, and filter as its key. An empty allowlist
selects zero rows.
The list is capped at 65,536 IDs. Broader predicates discard the entire
partial list and retain the complete native Substrait prefilter on the search
scanner. This bounds the C++ candidate list and preserves correctness. Profile
counters expose prefilter time, candidate count, cache hits, and fallback count.
### Scope and compatibility
- Automatic rewriting requires a direct Lance scan, an optional filter, a
projection, and a single search ordering key with LIMIT. It does not rewrite
across JOIN/EXISTS, add secondary sort keys, or support the internal-table
`SEARCH()` DSL.
- Vector rewriting supports physical Float32 fixed-size-list vectors and
constant query arrays. Exact `l2_distance()`, reversed vector ordering,
incompatible vector types, and vector queries with unsupported residual
predicates keep scalar execution.
- BM25 requires one supported top-level MATCH on a direct string column, a
literal query, the stored analyzer, and `ORDER BY score() DESC`. Remaining
scalar predicates must be fully convertible by the existing Lance predicate
converter. Unsupported scored-query shapes fail explicitly rather than silently
selecting incorrect candidates.
- The rewrite preserves bound snapshot/version metadata and output
expression IDs, retains column pruning, and disables SQL result caching for
rewritten searches.
- Existing explicit search TVF SQL-filter behavior is preserved.
### Validation
- FE reactor build and the relevant Maven test suite passed; after the final
compatibility guard, all 126 focused tests passed, including 12 new optimizer
tests.
- FE Checkstyle passed.
- Changed BE production and test translation units passed C++ syntax checks;
all affected C++ files passed clang-format 16 checks.
- Dependency download/MD5 and Foyer patch lifecycle tests passed.
- Rebuilt the exact pinned lance-c revision with the Foyer patch; native
runtime smoke tests against that library passed for high-bit UINT64 row IDs,
vector allowlists, empty allowlists, and prepared FTS allowlists with unchanged
BM25 scores.
- Added BE tests for query-local cache reuse, empty masks, and the
broad-filter fallback retaining the nearest row. Added an external-table
regression suite comparing automatic SQL with explicit TVFs and scalar distance
values.
- Full BE unit-test execution and cluster regression execution remain for
CI; they were not run locally.
### Release note
Support automatic native Lance search for eligible approximate vector Top-N
and MATCH/BM25 SQL on Lance external tables.
### Check List (For Author)
- Test
- [x] Regression test added (execution pending CI)
- [x] Unit Test added (FE passed; BE execution pending CI)
- [x] Manual native API smoke tests
- Behavior changed:
- [x] Yes. Eligible approximate vector queries use native search;
eligible scored MATCH queries become supported, as shown above.
- Does this need documentation?
- [x] Yes. The SQL forms and current boundaries are described above;
user documentation should accompany the feature release.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
