This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new cd673b8020e [feature](lance) Support distance bounds in vector_search
(#68815)
cd673b8020e is described below
commit cd673b8020ea4990c2109ef9adf6b4c7fd8ecf28
Author: Gabriel <[email protected]>
AuthorDate: Sat Oct 10 09:39:09 2026 +0800
[feature](lance) Support distance bounds in vector_search (#68815)
### What problem does this PR solve?
Issue Number: Closes #66493
`vector_search()` cannot currently restrict results by distance. This
adds optional `distance_lower_bound` (inclusive) and
`distance_upper_bound` (exclusive) parameters for single-vector
Radius/Range Search on Lance tables.
The FE validates finite FLOAT bounds and their order, includes the range
in EXPLAIN, and carries it through the external-search request. The BE
applies the bounds through the existing pinned lance-c API after
configuring the query vector, covering indexed and Flat Search splits,
including appended data outside index coverage. The existing
external-search schema version remains unchanged at 1.
Results retain the existing table columns and FLOAT `_distance`,
prefilter semantics, offset, and Top-K cap. Bounds can be bound through
prepared-statement parameters and are revalidated on each execution.
Multi-vector distance bounds are rejected.
Bounded index search is limited to explicitly identified IVF_FLAT
segments with both bounds present. Quantized or unknown index types and
single-sided ranges fall back to exact fragment scans, reported as
`DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement
quantization errors and the pinned index implementation’s finite
sentinels for absent bounds, at the cost of additional scanning.
Unbounded queries retain their index selection; indexed IVF_FLAT recall
still depends on probed partitions.
Standalone BM25 is already delivered; the distributed BM25 statistics
follow-up remains tracked separately in #67435.
### Release note
Lance `vector_search()` supports optional lower and upper distance
bounds for single-vector radius and range queries.
### Validation
- BE: rebuilt the current branch and passed all 68 `LanceTableReader*`
tests under ASAN, including exact SQ threshold crossing, the finite
FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage.
- FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`,
`LancePreparedSearchTest`, `LancePreparedStatementTest`, and
`LanceScanNodeTest`, including repeated bound binding, invalid
parameters, all index subtypes, and indexed/flat split selection.
- Native reproduction: the pinned indexed paths drop both the SQ
threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat
Search returns both. Added self-contained fixtures and reader coverage
for these cases.
- Native execution: 10 cases passed against the pinned lance-c revision,
covering indexed/Flat Search, appended unindexed data,
inclusive/exclusive bounds, empty results, prefiltering, and offset.
- FE Checkstyle and clang-format 16 checks passed.
- Added SQL regression coverage for range results, projected stored
columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy
syntax compilation passed; the SQL suite has not yet run against a live
Doris cluster and requires External Regression CI.
### Check List (For Author)
- Test
- [x] Regression test added (execution pending CI)
- [x] Unit Test
- [x] Manual native execution test
- Behavior changed:
- [x] Yes: optional distance bounds on single-vector searches; existing
unbounded queries retain their behavior.
- Does this need documentation?
- [x] Yes: bilingual 4.x Lance Catalog documentation:
https://github.com/apache/doris-website/pull/4200.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/format_v2/table/lance_reader.cpp | 33 ++++
.../auxiliary.idx | Bin 0 -> 3254 bytes
.../9634c881-df64-46a5-95dc-2468928008a5/index.idx | Bin 0 -> 329 bytes
.../0-388c379e-6e57-4a48-a567-603ce868bdea.txn | Bin 0 -> 205 bytes
.../1-831523ce-b62e-46b0-bcd2-501920af09b8.txn | Bin 0 -> 376 bytes
.../2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn | Bin 0 -> 122 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 824 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 997 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 486 bytes
.../_versions/latest_version_hint.json | 1 +
...100000101111111a29c2749f892029e3946d88007.lance | Bin 0 -> 661 bytes
...000101110010011f2593f448ba30e92dc54b90567.lance | Bin 0 -> 3100 bytes
.../auxiliary.idx | Bin 0 -> 3255 bytes
.../76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx | Bin 0 -> 330 bytes
.../0-ec982602-f60a-425e-8b31-f77179548301.txn | Bin 0 -> 205 bytes
.../1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn | Bin 0 -> 378 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1001 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 487 bytes
.../_versions/latest_version_hint.json | 1 +
...000110101111101bccc4b4c19af77197f1a92b69c.lance | Bin 0 -> 3100 bytes
.../auxiliary.idx | Bin 0 -> 2805 bytes
.../16361222-c72d-4428-9826-beeb3bfd3667/index.idx | Bin 0 -> 327 bytes
.../0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn | Bin 0 -> 205 bytes
.../1-f4a6b175-3626-4561-817e-c45c857597fa.txn | Bin 0 -> 407 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1059 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 487 bytes
.../_versions/latest_version_hint.json | 1 +
...1010101000001014167dc46e9aa87a5e01e036b2a.lance | Bin 0 -> 4892 bytes
.../table/lance/distance_range_fixture.md | 93 ++++++++++++
be/test/format_v2/table/lance_reader_test.cpp | 167 +++++++++++++++++++++
.../datasource/lance/source/LanceScanNode.java | 7 +
.../datasource/lance/source/LanceScanPlan.java | 2 +
.../datasource/lance/source/LanceScanPlanner.java | 12 ++
.../doris/nereids/parser/LogicalPlanBuilder.java | 4 +-
.../VectorSearchTableValuedFunction.java | 44 +++++-
.../datasource/lance/source/LanceScanNodeTest.java | 40 +++++
.../tablefunction/LancePreparedSearchTest.java | 105 +++++++++++++
.../VectorSearchTableValuedFunctionTest.java | 47 ++++++
gensrc/thrift/PlanNodes.thrift | 3 +
.../test_lance_vector_search_distance_range.groovy | 82 ++++++++++
40 files changed, 638 insertions(+), 4 deletions(-)
diff --git a/be/src/format_v2/table/lance_reader.cpp
b/be/src/format_v2/table/lance_reader.cpp
index 73d68332f49..c445d795d94 100644
--- a/be/src/format_v2/table/lance_reader.cpp
+++ b/be/src/format_v2/table/lance_reader.cpp
@@ -448,6 +448,28 @@ Status
LanceTableReader::_validate_external_search_request() const {
static_cast<int>(query_vector.element_type));
}
const bool multi_vector = query_vector.__isset.num_vectors;
+ const bool has_distance_range =
+ vector.__isset.distance_lower_bound ||
vector.__isset.distance_upper_bound;
+ if (has_distance_range && multi_vector) {
+ return Status::NotSupported(
+ "Distance bounds are not supported for Lance multi-vector
search");
+ }
+ // Validate doubles before narrowing: non-finite or out-of-range
values cannot reach the C API.
+ const auto valid_bound = [](double bound) {
+ return std::isfinite(bound) && std::abs(bound) <=
std::numeric_limits<float>::max();
+ };
+ if ((vector.__isset.distance_lower_bound &&
!valid_bound(vector.distance_lower_bound)) ||
+ (vector.__isset.distance_upper_bound &&
!valid_bound(vector.distance_upper_bound))) {
+ return Status::InvalidArgument("Lance distance bounds must be
finite FLOAT values");
+ }
+ if (vector.__isset.distance_lower_bound &&
vector.__isset.distance_upper_bound &&
+ static_cast<float>(vector.distance_lower_bound) >=
+ static_cast<float>(vector.distance_upper_bound)) {
+ return Status::InvalidArgument(
+ "distance_lower_bound must be less than
distance_upper_bound at FLOAT "
+ "precision");
+ }
+
// The optional count distinguishes a query matrix, including a
one-row matrix.
if (multi_vector && query_vector.num_vectors <= 0) {
return Status::InvalidArgument(
@@ -1177,6 +1199,17 @@ Status
LanceTableReader::_configure_vector_search(LanceScanner* scanner,
}
}
+ if (vector.__isset.distance_lower_bound ||
vector.__isset.distance_upper_bound) {
+ const float lower = static_cast<float>(vector.distance_lower_bound);
+ const float upper = static_cast<float>(vector.distance_upper_bound);
+ // nearest() clears the range; install bounds only after setting the
query vector.
+ if (lance_scanner_set_distance_range(
+ scanner, vector.__isset.distance_lower_bound ? &lower :
nullptr,
+ vector.__isset.distance_upper_bound ? &upper : nullptr) !=
0) {
+ return lance_error("set Lance vector distance range");
+ }
+ }
+
// Exact refinement validates stored elements and gives indexed and
unindexed candidates
// the same row-level score before either path truncates its results.
if (query.__isset.num_vectors && lance_scanner_set_refine_factor(scanner,
1) != 0) {
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/auxiliary.idx
b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/auxiliary.idx
new file mode 100644
index 00000000000..f6ecf98bb6c
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/auxiliary.idx
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/index.idx
b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/index.idx
new file mode 100644
index 00000000000..d7646e4098c
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/index.idx
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/0-388c379e-6e57-4a48-a567-603ce868bdea.txn
b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/0-388c379e-6e57-4a48-a567-603ce868bdea.txn
new file mode 100644
index 00000000000..6643a15e80a
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/0-388c379e-6e57-4a48-a567-603ce868bdea.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/1-831523ce-b62e-46b0-bcd2-501920af09b8.txn
b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/1-831523ce-b62e-46b0-bcd2-501920af09b8.txn
new file mode 100644
index 00000000000..949b47f39d9
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/1-831523ce-b62e-46b0-bcd2-501920af09b8.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn
b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn
new file mode 100644
index 00000000000..afebdade12d
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551612.manifest
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551612.manifest
new file mode 100644
index 00000000000..7ed8e3d926b
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551612.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551613.manifest
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551613.manifest
new file mode 100644
index 00000000000..ba5cc92bc12
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551613.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551614.manifest
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551614.manifest
new file mode 100644
index 00000000000..8f32f9b807f
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551614.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/latest_version_hint.json
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/latest_version_hint.json
new file mode 100644
index 00000000000..f93d3984472
--- /dev/null
+++
b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/latest_version_hint.json
@@ -0,0 +1 @@
+{"version":3}
\ No newline at end of file
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/data/011111011100000101111111a29c2749f892029e3946d88007.lance
b/be/test/format_v2/table/lance/data/distance_range.lance/data/011111011100000101111111a29c2749f892029e3946d88007.lance
new file mode 100644
index 00000000000..52f7dbf29cc
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/data/011111011100000101111111a29c2749f892029e3946d88007.lance
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range.lance/data/100110001000101110010011f2593f448ba30e92dc54b90567.lance
b/be/test/format_v2/table/lance/data/distance_range.lance/data/100110001000101110010011f2593f448ba30e92dc54b90567.lance
new file mode 100644
index 00000000000..53e1eec28d0
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range.lance/data/100110001000101110010011f2593f448ba30e92dc54b90567.lance
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/auxiliary.idx
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/auxiliary.idx
new file mode 100644
index 00000000000..3ca57b8d6de
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/auxiliary.idx
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx
new file mode 100644
index 00000000000..446e4f4152b
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/0-ec982602-f60a-425e-8b31-f77179548301.txn
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/0-ec982602-f60a-425e-8b31-f77179548301.txn
new file mode 100644
index 00000000000..72e846c8998
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/0-ec982602-f60a-425e-8b31-f77179548301.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn
new file mode 100644
index 00000000000..57cf3b8d551
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551613.manifest
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551613.manifest
new file mode 100644
index 00000000000..d3526033b05
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551613.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551614.manifest
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551614.manifest
new file mode 100644
index 00000000000..9aebc63bd80
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551614.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/latest_version_hint.json
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/latest_version_hint.json
new file mode 100644
index 00000000000..218abba1699
--- /dev/null
+++
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/latest_version_hint.json
@@ -0,0 +1 @@
+{"version":2}
\ No newline at end of file
diff --git
a/be/test/format_v2/table/lance/data/distance_range_dot.lance/data/110000101000110101111101bccc4b4c19af77197f1a92b69c.lance
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/data/110000101000110101111101bccc4b4c19af77197f1a92b69c.lance
new file mode 100644
index 00000000000..b72e061da89
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_dot.lance/data/110000101000110101111101bccc4b4c19af77197f1a92b69c.lance
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/auxiliary.idx
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/auxiliary.idx
new file mode 100644
index 00000000000..1483b3a7c8e
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/auxiliary.idx
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/index.idx
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/index.idx
new file mode 100644
index 00000000000..79948a808fe
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/index.idx
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn
new file mode 100644
index 00000000000..6cd36ac9668
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/1-f4a6b175-3626-4561-817e-c45c857597fa.txn
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/1-f4a6b175-3626-4561-817e-c45c857597fa.txn
new file mode 100644
index 00000000000..b2616f9b6e6
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/1-f4a6b175-3626-4561-817e-c45c857597fa.txn
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551613.manifest
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551613.manifest
new file mode 100644
index 00000000000..3a4c998b412
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551613.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551614.manifest
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551614.manifest
new file mode 100644
index 00000000000..97194c3229a
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551614.manifest
differ
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/latest_version_hint.json
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/latest_version_hint.json
new file mode 100644
index 00000000000..218abba1699
--- /dev/null
+++
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/latest_version_hint.json
@@ -0,0 +1 @@
+{"version":2}
\ No newline at end of file
diff --git
a/be/test/format_v2/table/lance/data/distance_range_sq.lance/data/0000100001010101000001014167dc46e9aa87a5e01e036b2a.lance
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/data/0000100001010101000001014167dc46e9aa87a5e01e036b2a.lance
new file mode 100644
index 00000000000..ac784824743
Binary files /dev/null and
b/be/test/format_v2/table/lance/data/distance_range_sq.lance/data/0000100001010101000001014167dc46e9aa87a5e01e036b2a.lance
differ
diff --git a/be/test/format_v2/table/lance/distance_range_fixture.md
b/be/test/format_v2/table/lance/distance_range_fixture.md
new file mode 100644
index 00000000000..d1f3723304b
--- /dev/null
+++ b/be/test/format_v2/table/lance/distance_range_fixture.md
@@ -0,0 +1,93 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Distance range fixture
+
+Synthetic single-vector data generated with `pylance==7.0.0`, using Lance data
+format 2.2 and an IVF_FLAT V3 index. No external service is required to read
it.
+Rows 0–127 are indexed in fragment 0. Rows 128–130 were appended after indexing
+in fragment 1. For query `[127, 0, 0]`, range `[1, 4)` returns rows 126 and
128,
+which exercises both index coverage and the unindexed tail.
+
+Regenerate from the repository root:
+
+```python
+import lance
+import pyarrow as pa
+
+path = "be/test/format_v2/table/lance/data/distance_range.lance"
+schema = pa.schema([
+ pa.field("row_id", pa.int64(), nullable=False),
+ pa.field("embedding", pa.list_(pa.float32(), 3), nullable=False),
+])
+
+def batch(ids):
+ return pa.Table.from_pydict({
+ "row_id": list(ids),
+ "embedding": [[float(i), 0.0, 0.0] for i in ids],
+ }, schema=schema)
+
+dataset = lance.write_dataset(batch(range(128)), path, mode="overwrite",
+ data_storage_version="2.2")
+dataset.create_index("embedding", index_type="IVF_FLAT", num_partitions=1,
+ name="embedding_idx", index_file_version="V3")
+lance.write_dataset(batch(range(128, 131)), path, mode="append",
+ data_storage_version="2.2")
+```
+
+Delete the previous generated dataset directory before regenerating to keep the
+fixture free of obsolete files. Filenames and index UUIDs may change; tests
discover
+them from the manifest.
+
+## Range fallback fixtures
+
+`distance_range_sq.lance` contains vectors `[i, 0, 0]` for integers 0 through
255,
+followed by row 256 with `[0.49, 0, 0]`, and an IVF_SQ V3 index. With query
+`[0.51, 0, 0]`, row 256 has squared L2 distance approximately `0.0004`, inside
+`[0.0001, 0.001)`. Its quantized distance is zero, so applying the lower bound
+before refinement incorrectly discards it.
+
+`distance_range_dot.lance` contains row 0 with `[1, 0, 0]` and 127 rows with
+`[0.5, 0, 0]`, and an IVF_FLAT V3 DOT index. Query `[-FLT_MAX, 0, 0]` with only
+an inclusive lower bound of `FLT_MAX` must return row 0. Replacing the absent
+upper bound with an exclusive `FLT_MAX` incorrectly discards it.
+
+These fixtures exercise fragment splits without index UUIDs, as planned by the
+FE for unsafe indexed range searches, with both `use_index=true` and `false`.
+They use the same schema and writer versions as the fixture above:
+
+```python
+from pathlib import Path
+
+root = Path("be/test/format_v2/table/lance/data")
+for name, values, index_type, metric in [
+ ("distance_range_sq", [[float(i), 0.0, 0.0] for i in range(256)]
+ + [[0.49, 0.0, 0.0]], "IVF_SQ", "l2"),
+ ("distance_range_dot", [[1.0, 0.0, 0.0]]
+ + [[0.5, 0.0, 0.0] for _ in range(127)], "IVF_FLAT", "dot"),
+]:
+ data = pa.Table.from_pydict({
+ "row_id": list(range(len(values))), "embedding": values,
+ }, schema=schema)
+ dataset = lance.write_dataset(data, str(root / (name + ".lance")),
+ data_storage_version="2.2")
+ dataset.create_index("embedding", index_type=index_type, metric=metric,
+ num_partitions=1, name="embedding_idx",
+ index_file_version="V3")
+```
diff --git a/be/test/format_v2/table/lance_reader_test.cpp
b/be/test/format_v2/table/lance_reader_test.cpp
index d13665f2998..aba7cfee95b 100644
--- a/be/test/format_v2/table/lance_reader_test.cpp
+++ b/be/test/format_v2/table/lance_reader_test.cpp
@@ -1600,6 +1600,173 @@ TEST(LanceTableReaderVectorSearchTest,
RejectsInvalidQueryParallelismBeforeOpeni
EXPECT_NE(std::string::npos, status.to_string().find("query_parallelism"));
}
+TEST(LanceTableReaderVectorSearchTest,
ValidatesDistanceRangeBeforeDatasetAccess) {
+ TQueryGlobals globals;
+ RuntimeState state(globals);
+ const Columns columns {projected_column("_distance", TYPE_FLOAT, true)};
+ for (int mode = 0; mode < 6; ++mode) {
+ SCOPED_TRACE(mode);
+ auto params = make_float32_vector_search_params({0.0F, 0.0F, 0.0F}, 2,
0);
+ auto& request = params.lance_scan_params.external_search_request;
+ auto& vector = request.search_query.vector_search;
+ vector.__set_distance_lower_bound(1.0);
+ vector.__set_distance_upper_bound(2.0);
+ if (mode == 0)
vector.__set_distance_upper_bound(std::numeric_limits<double>::infinity());
+ if (mode == 1)
vector.__set_distance_lower_bound(std::numeric_limits<double>::quiet_NaN());
+ if (mode == 2)
vector.__set_distance_upper_bound(std::numeric_limits<double>::max());
+ if (mode == 3) vector.__set_distance_upper_bound(1.0);
+ if (mode == 4) vector.__set_distance_upper_bound(1.00000001);
+ if (mode == 5) vector.query_vector.__set_num_vectors(1);
+ RuntimeProfile profile("invalid_distance_range");
+ LanceTableReader reader;
+ const auto status = init_reader(&reader, columns, &state, &profile,
¶ms);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(std::string::npos,
+ status.to_string().find(mode == 5 ? "multi-vector" :
"distance"));
+ }
+}
+
+TEST(LanceTableReaderVectorSearchTest, AppliesDistanceBoundsAndOffset) {
+ const std::filesystem::path uri =
"./be/test/format_v2/table/lance/data/all_types.lance";
+ LanceFixtureInfo fixture;
+ ASSERT_TRUE(get_fixture_info(uri, &fixture).ok());
+ const Columns columns {projected_column("row_id", TYPE_BIGINT, false),
+ projected_column("_distance", TYPE_FLOAT, true)};
+ TQueryGlobals globals;
+ RuntimeState state(globals);
+ for (int mode = 0; mode < 5; ++mode) {
+ SCOPED_TRACE(mode);
+ auto params = make_float32_vector_search_params({0.0F, 0.0F, 0.0F},
10, mode == 4 ? 1 : 0);
+ auto& request = params.lance_scan_params.external_search_request;
+ auto& vector = request.search_query.vector_search;
+ if (mode != 1) vector.__set_distance_lower_bound(1.0);
+ if (mode != 2) vector.__set_distance_upper_bound(mode == 3 ? 1.1 :
8.25);
+ RuntimeProfile profile("distance_range");
+ LanceTableReader reader;
+ ASSERT_TRUE(init_reader(&reader, columns, &state, &profile,
¶ms).ok());
+ ASSERT_TRUE(prepare_fixture(&reader, uri, fixture,
fixture.fragment_ids).ok());
+ Block block;
+ add_output_columns(&block, columns);
+ const auto rows = read_vector_search_rows(&reader, &block);
+ if (mode == 4) {
+ EXPECT_TRUE(rows.empty());
+ } else {
+ ASSERT_FALSE(rows.empty());
+ EXPECT_EQ(mode == 1 ? 1 : 2, rows[0].first);
+ for (const auto& [id, distance] : rows) {
+ if (mode != 1) EXPECT_GE(distance, 1.0F);
+ if (mode != 2) EXPECT_LT(distance, mode == 3 ? 1.1F : 8.25F);
+ }
+ if (mode == 0 || mode == 3) EXPECT_EQ(1U, rows.size());
+ }
+ ASSERT_TRUE(reader.close().ok());
+ }
+}
+
+TEST(LanceTableReaderVectorSearchTest,
DistanceRangeFallbackPreservesExactScores) {
+ const Columns columns {projected_column("row_id", TYPE_BIGINT, false),
+ projected_column("_distance", TYPE_FLOAT, true)};
+ TQueryGlobals globals;
+ RuntimeState state(globals);
+ for (const bool dot : {false, true}) {
+ const std::filesystem::path uri =
+ dot ?
"./be/test/format_v2/table/lance/data/distance_range_dot.lance"
+ :
"./be/test/format_v2/table/lance/data/distance_range_sq.lance";
+ LanceFixtureInfo fixture;
+ ASSERT_TRUE(get_fixture_info(uri, &fixture).ok());
+ for (const bool use_index : {false, true}) {
+ for (const bool filtered : {false, true}) {
+ SCOPED_TRACE(std::to_string(dot) + "/" +
std::to_string(use_index) + "/" +
+ std::to_string(filtered));
+ const float max = std::numeric_limits<float>::max();
+ auto params = make_float32_vector_search_params(
+ dot ? std::array<float, 3> {-max, 0.0F, 0.0F}
+ : std::array<float, 3> {0.51F, 0.0F, 0.0F},
+ 300, 0,
+ filtered ? std::optional<std::string>("row_id >= 0") :
std::nullopt);
+ auto& request =
params.lance_scan_params.external_search_request;
+ auto& vector = request.search_query.vector_search;
+ vector.__set_distance_lower_bound(dot ? max : 0.0001);
+ if (dot) {
+ vector.__set_metric(TVectorMetric::DOT_PRODUCT);
+ } else {
+ vector.__set_distance_upper_bound(0.001);
+ }
+ request.vector_search_options.__set_use_index(use_index);
+ request.vector_search_options.__set_refine_factor(1);
+ request.vector_search_options.__set_query_parallelism(4);
+ // Range fallback splits have no index UUIDs even when
use_index was requested.
+ // SQ quantization must not drop 0.0004, nor an absent upper
bound drop FLT_MAX.
+ RuntimeProfile profile("distance_range_fallback");
+ LanceTableReader reader;
+ ASSERT_TRUE(init_reader(&reader, columns, &state, &profile,
¶ms).ok());
+ ASSERT_TRUE(prepare_fixture(&reader, uri, fixture,
fixture.fragment_ids).ok());
+ Block block;
+ add_output_columns(&block, columns);
+ const auto rows = read_vector_search_rows(&reader, &block);
+ ASSERT_EQ(1U, rows.size());
+ EXPECT_EQ(dot ? 0 : 256, rows[0].first);
+ if (dot) {
+ EXPECT_FLOAT_EQ(max, rows[0].second);
+ } else {
+ EXPECT_NEAR(0.0004F, rows[0].second, 1e-8F);
+ }
+ ASSERT_TRUE(reader.close().ok());
+ }
+ }
+ }
+}
+
+TEST(LanceTableReaderVectorSearchTest,
DistanceRangeCoversIndexedAndAppendedFragments) {
+ const std::filesystem::path uri =
"./be/test/format_v2/table/lance/data/distance_range.lance";
+ LanceFixtureInfo fixture;
+ ASSERT_TRUE(get_fixture_info(uri, &fixture).ok());
+ ASSERT_EQ(2U, fixture.fragment_ids.size());
+ std::vector<std::string> segments;
+ ASSERT_TRUE(get_index_segment_uuids(uri, "embedding_idx", &segments).ok());
+ ASSERT_EQ(1U, segments.size());
+ const Columns columns {projected_column("row_id", TYPE_BIGINT, false),
+ projected_column("_distance", TYPE_FLOAT, true)};
+ TQueryGlobals globals;
+ RuntimeState state(globals);
+ for (const bool use_index : {false, true}) {
+ for (const bool filtered : {false, true}) {
+ std::vector<int64_t> ids;
+ for (size_t split = 0; split < fixture.fragment_ids.size();
++split) {
+ SCOPED_TRACE(std::to_string(use_index) + "/" +
std::to_string(filtered) + "/" +
+ std::to_string(split));
+ auto params = make_float32_vector_search_params(
+ {127.0F, 0.0F, 0.0F}, 10, 0,
+ filtered ? std::optional<std::string>("row_id >= 127")
: std::nullopt);
+ auto& request =
params.lance_scan_params.external_search_request;
+
request.search_query.vector_search.__set_distance_lower_bound(1.0);
+
request.search_query.vector_search.__set_distance_upper_bound(4.0);
+ const bool indexed_split = use_index && split == 0;
+ request.vector_search_options.__set_use_index(indexed_split);
+ request.vector_search_options.__set_nprobes(1);
+ auto range = make_lance_range(uri, fixture.version,
{fixture.fragment_ids[split]});
+ if (indexed_split) {
+
range.table_format_params.lance_params.__set_index_segment_uuids(segments);
+ }
+ RuntimeProfile profile("mixed_distance_range");
+ LanceTableReader reader;
+ ASSERT_TRUE(init_reader(&reader, columns, &state, &profile,
¶ms).ok());
+ ASSERT_TRUE(prepare_range(&reader, range).ok());
+ Block block;
+ add_output_columns(&block, columns);
+ for (const auto& [id, distance] :
read_vector_search_rows(&reader, &block)) {
+ ids.push_back(id);
+ EXPECT_FLOAT_EQ(1.0F, distance);
+ }
+ ASSERT_TRUE(reader.close().ok());
+ }
+ std::ranges::sort(ids);
+ EXPECT_EQ(filtered ? std::vector<int64_t> {128} :
std::vector<int64_t>({126, 128}),
+ ids);
+ }
+ }
+}
+
TEST(LanceTableReaderVectorSearchTest, QueryParallelismPreservesResults) {
const std::filesystem::path uri =
"./be/test/format_v2/table/lance/data/all_types.lance";
LanceFixtureInfo fixture;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
index c572234f027..4ac720cad4d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
@@ -359,6 +359,13 @@ public class LanceScanNode extends FileQueryScanNode {
result.append(prefix).append("lanceQueryParallelism=")
.append(externalSearchRequest.getVectorSearchOptions().getQueryParallelism()).append("\n");
}
+ if (vector.isSetDistanceLowerBound() ||
vector.isSetDistanceUpperBound()) {
+ result.append(prefix).append("lanceDistanceRange=[")
+ .append(vector.isSetDistanceLowerBound() ?
vector.getDistanceLowerBound() : "-inf")
+ .append(", ")
+ .append(vector.isSetDistanceUpperBound() ?
vector.getDistanceUpperBound() : "+inf")
+ .append(")\n");
+ }
result.append(prefix).append("lanceMetric=")
.append(vector.isSetMetric()
?
VectorSearchTableValuedFunction.metricName(vector.getMetric()) : "default")
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java
index d2259c697ed..b4968f1d67f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java
@@ -65,6 +65,8 @@ final class LanceScanPlan {
NO_MATCH,
/** The selected index has a different or unknown metric. */
METRIC_MISMATCH,
+ /** The pinned index search cannot safely evaluate this distance
range. */
+ DISTANCE_RANGE_FALLBACK,
/** At least one segment has no fragment coverage metadata. */
UNKNOWN_COVERAGE,
/** All segment coverage lies outside the selected snapshot's visible
fragments. */
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java
index 2d97bc802bf..96925ff2399 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java
@@ -35,6 +35,8 @@ import org.apache.doris.thrift.TVectorMetric;
import org.apache.doris.thrift.TVectorSearchOptions;
import org.apache.doris.thrift.TVectorSearchParams;
+import org.lance.index.IndexType;
+
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
@@ -227,6 +229,16 @@ final class LanceScanPlanner {
vectorIndexStatus = VectorIndexStatus.METRIC_MISMATCH;
continue;
}
+ boolean hasLower = vectorSearchParam.isSetDistanceLowerBound();
+ boolean hasUpper = vectorSearchParam.isSetDistanceUpperBound();
+ // Quantized indexes apply bounds before exact refinement and can
discard valid rows.
+ // The pinned parallel IVF_FLAT path also replaces absent bounds
with finite sentinels.
+ // Only a two-sided IVF_FLAT range is safe; unknown index types
must use exact scans.
+ if ((hasLower || hasUpper) && (!hasLower || !hasUpper ||
matchingSegments.stream()
+ .anyMatch(segment -> segment.getIndexType() !=
IndexType.IVF_FLAT))) {
+ vectorIndexStatus = VectorIndexStatus.DISTANCE_RANGE_FALLBACK;
+ continue;
+ }
Optional<LanceSplitBuilder> indexPlan = planIndexSegments(
metadata, matchingSegments, visibleFragments,
IndexCoveragePolicy.VECTOR_FALLBACK);
if (!indexPlan.isPresent()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 2830de1f61d..afe313a8c78 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -2884,7 +2884,9 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
throw new AnalysisException("Duplicate vector_search
property: " + key);
}
if (argument.value.constant() instanceof
DorisParser.PlaceholderContext) {
- if (!ImmutableSet.of("query_vector", "top_k",
"offset", "filter").contains(key)) {
+ // Bounds change the search results, but never the
prepared result schema.
+ if (!ImmutableSet.of("query_vector", "top_k",
"offset", "filter",
+ "distance_lower_bound",
"distance_upper_bound").contains(key)) {
throw new AnalysisException("vector_search
property '" + key
+ "' must be constant in a prepared
statement");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
index 2e9e2374b7f..d9d121924fd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java
@@ -64,10 +64,13 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
private static final String NPROBES = "nprobes";
private static final String REFINE_FACTOR = "refine_factor";
private static final String EF = "ef";
+ private static final String DISTANCE_LOWER_BOUND = "distance_lower_bound";
+ private static final String DISTANCE_UPPER_BOUND = "distance_upper_bound";
private static final String USE_INDEX = "use_index";
private static final Set<String> PROPERTIES = ImmutableSet.of(
TABLE, COLUMN, QUERY_VECTOR, TOP_K, OFFSET, METRIC, FILTER,
- NPROBES, REFINE_FACTOR, EF, USE_INDEX, QUERY_PARALLELISM, VERSION,
TIMESTAMP, TAG, BRANCH);
+ NPROBES, REFINE_FACTOR, EF, USE_INDEX, QUERY_PARALLELISM,
+ VERSION, TIMESTAMP, TAG, BRANCH, DISTANCE_LOWER_BOUND,
DISTANCE_UPPER_BOUND);
public VectorSearchTableValuedFunction(Map<String, String> properties)
throws AnalysisException {
@@ -82,6 +85,8 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
private static PreparedSearch prepare(Map<String, String> properties,
boolean deferQueryVector)
throws AnalysisException {
Map<String, String> params = normalizeProperties(properties,
PROPERTIES, NAME);
+ TVectorSearchParams vectorParams = parseDistanceBounds(params);
+ boolean hasDistanceRange = vectorParams.isSetDistanceLowerBound() ||
vectorParams.isSetDistanceUpperBound();
boolean useIndex = !params.containsKey(USE_INDEX)
|| parseBoolean(params.get(USE_INDEX), USE_INDEX);
CommonSearch common = prepareCommon(params, NAME,
@@ -89,6 +94,10 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
Field vectorField = requireSearchColumn(
common.metadata().getSchema(), required(params, COLUMN, NAME),
"vector");
+ // Reject matrices at PREPARE as well as EXECUTE; lance-c only
supports single-vector ranges.
+ if (hasDistanceRange && vectorField.getType() instanceof
ArrowType.List) {
+ throw new AnalysisException("Distance bounds are not supported for
Lance multi-vector search");
+ }
int vectorFieldId = useIndex
? requireLanceFieldId(common.metadata(), vectorField) : -1;
// PREPARE needs the table schema, but the vector is supplied only at
EXECUTE.
@@ -100,8 +109,7 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
vectorField, required(params, QUERY_VECTOR, NAME));
}
- TVectorSearchParams vectorParams = new TVectorSearchParams()
- .setColumn(vectorField.getName())
+ vectorParams.setColumn(vectorField.getName())
.setQueryVector(queryVector)
.setTopK(common.topK())
.setOffset(common.offset());
@@ -126,6 +134,36 @@ public class VectorSearchTableValuedFunction extends
LanceExternalSearchTableVal
common, vectorFieldId, searchRequest, DISTANCE_COLUMN, "vector
search");
}
+ @VisibleForTesting
+ static TVectorSearchParams parseDistanceBounds(Map<String, String> params)
throws AnalysisException {
+ TVectorSearchParams vector = new TVectorSearchParams();
+ if (params.containsKey(DISTANCE_LOWER_BOUND)) {
+
vector.setDistanceLowerBound(parseDistanceBound(params.get(DISTANCE_LOWER_BOUND),
DISTANCE_LOWER_BOUND));
+ }
+ if (params.containsKey(DISTANCE_UPPER_BOUND)) {
+
vector.setDistanceUpperBound(parseDistanceBound(params.get(DISTANCE_UPPER_BOUND),
DISTANCE_UPPER_BOUND));
+ }
+ // Compare after FLOAT rounding, matching the native API even for
nearly equal inputs.
+ if (vector.isSetDistanceLowerBound() &&
vector.isSetDistanceUpperBound()
+ && vector.getDistanceLowerBound() >=
vector.getDistanceUpperBound()) {
+ throw new AnalysisException(
+ "distance_lower_bound must be less than
distance_upper_bound at FLOAT precision");
+ }
+ return vector;
+ }
+
+ private static float parseDistanceBound(String value, String property)
throws AnalysisException {
+ try {
+ float bound = Float.parseFloat(value);
+ if (Float.isFinite(bound)) {
+ return bound;
+ }
+ } catch (NumberFormatException | NullPointerException ignored) {
+ // Report the SQL property rather than leaking a Java parsing
exception.
+ }
+ throw new AnalysisException("'" + property + "' must be a finite
FLOAT");
+ }
+
@VisibleForTesting
static TVectorSearchOptions buildVectorSearchOptions(
Map<String, String> params, boolean useIndex) throws
AnalysisException {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
index bd555d1dab3..32173603c8f 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java
@@ -605,6 +605,46 @@ public class LanceScanNodeTest {
assertSplit(splits.get(3), 4, 8, 63);
}
+ @Test
+ public void testDistanceRangesOnlyUseSafeIndexPaths() throws Exception {
+ for (IndexType type : IndexType.values()) {
+ if (type.getValue() < IndexType.VECTOR.getValue()) {
+ continue;
+ }
+ UUID segment =
UUID.fromString("11111111-2222-3333-4444-555555555555");
+ LanceTableMetadata metadata =
LanceTableMetadata.createSnapshotWithIndexes(
+ new LanceTableAccess("s3://bucket/table.lance",
Collections.emptyMap()),
+ 42, vectorSchema(), Arrays.asList(new LanceFragmentInfo(1,
8, 8),
+ new LanceFragmentInfo(2, 7, 7)),
+ Collections.singletonMap("vector", 9),
+ Collections.singletonList(new
LanceIndexSegmentInfo(segment, "vector_idx",
+ Collections.singletonList(9),
Collections.singletonList(1L), type, "L2")));
+ for (int bounds = 0; bounds < 4; bounds++) {
+ TExternalSearchRequest request = vectorSearchRequest(5, 0);
+ if ((bounds & 1) != 0) {
+
request.getSearchQuery().getVectorSearch().setDistanceLowerBound(1);
+ }
+ if ((bounds & 2) != 0) {
+
request.getSearchQuery().getVectorSearch().setDistanceUpperBound(4);
+ }
+ request.setVectorSearchOptions(new
TVectorSearchOptions().setUseIndex(true).setQueryParallelism(4));
+ LanceScanNode node = newSearchNode(metadata, request);
+ List<Split> splits = node.getSplits(2);
+ boolean indexed = bounds == 0 || (bounds == 3 && type ==
IndexType.IVF_FLAT);
+ String status = indexed ? "USED" : "DISTANCE_RANGE_FALLBACK";
+ Assert.assertTrue(type + "/" + bounds,
node.getNodeExplainString("", TExplainLevel.NORMAL)
+ .contains("lanceVectorIndexStatus=" + status));
+ Assert.assertEquals(2, splits.size());
+ if (indexed) {
+ assertIndexSplit(splits.get(0), segment,
Collections.singletonList(1L), 8, 100);
+ } else {
+ assertSplit(splits.get(0), 1, 8, 100);
+ }
+ assertSplit(splits.get(1), 2, 8, 88);
+ }
+ }
+ }
+
@Test
public void
testExternalSearchUsesOneSplitPerIndexSegmentAndKeepsUnindexedFragments()
throws Exception {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java
index 074de75998f..7c2a2361412 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java
@@ -62,9 +62,114 @@ import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
+import java.util.Map;
import java.util.concurrent.atomic.AtomicLong;
public class LancePreparedSearchTest {
+ @ParameterizedTest
+ @ValueSource(strings = {"'distance_lower_bound'=?,
'distance_upper_bound'=?",
+ "'distance_lower_bound'=?", "'distance_upper_bound'=?",
+ "'distance_lower_bound'='-1', 'distance_upper_bound'=?",
+ "'distance_lower_bound'=?, 'distance_upper_bound'='4'"})
+ public void testRepeatedBindingUsesFreshDistanceBounds(String bounds) {
+ ConnectContext previous = ConnectContext.get();
+ ConnectContext context = MemoTestUtils.createConnectContext();
+ LanceExternalTable table = mockTable(new AtomicLong(42));
+ try (MockedStatic<LanceExternalSearchTableValuedFunction> lookup =
Mockito.mockStatic(
+ LanceExternalSearchTableValuedFunction.class,
Mockito.CALLS_REAL_METHODS)) {
+ lookup.when(() ->
LanceExternalSearchTableValuedFunction.findLanceExternalTable(
+ Mockito.any(TableName.class))).thenReturn(table);
+ Pair<LogicalPlan, StatementContext> parsed = new
NereidsParser().parseMultiple(
+ "select * from vector_search('table'='catalog.db.items',
'column'='embedding', "
+ + "'query_vector'='[1,2]', " + bounds +
")").get(0);
+ List<Placeholder> parameters = parsed.second.getPlaceholders();
+ StatementContext prepare =
MemoTestUtils.createStatementContext(context, "");
+ prepare.setPrepareStage(true);
+ analyze(parsed.first, prepare);
+ for (int iteration = 0; iteration < 2; iteration++) {
+ StatementContext execute =
MemoTestUtils.createStatementContext(context, "");
+ for (int i = 0; i < parameters.size(); i++) {
+
execute.getIdToPlaceholderRealExpr().put(parameters.get(i).getPlaceholderId(),
+ new StringLiteral(Integer.toString(iteration +
i)));
+ }
+ org.apache.doris.thrift.TVectorSearchParams vector =
analyze(parsed.first, execute)
+ .getSearchRequest().getSearchQuery().getVectorSearch();
+
Assertions.assertEquals(bounds.contains("distance_lower_bound"),
vector.isSetDistanceLowerBound());
+
Assertions.assertEquals(bounds.contains("distance_upper_bound"),
vector.isSetDistanceUpperBound());
+ if (vector.isSetDistanceLowerBound()) {
+ Assertions.assertEquals(bounds.contains("='-1'") ? -1.0 :
iteration,
+ vector.getDistanceLowerBound());
+ }
+ if (vector.isSetDistanceUpperBound()) {
+ Assertions.assertEquals(bounds.contains("='4'") ? 4.0 :
iteration + parameters.size() - 1,
+ vector.getDistanceUpperBound());
+ }
+ // Failed EXECUTEs must not poison the reusable unbound plan
or retain old bounds.
+ for (String invalid : new String[] {"NaN", "Infinity", "1e39",
"not-a-number"}) {
+
execute.getIdToPlaceholderRealExpr().put(parameters.get(0).getPlaceholderId(),
+ new StringLiteral(invalid));
+
Assertions.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class,
+ () -> analyze(parsed.first, execute));
+ }
+
execute.getIdToPlaceholderRealExpr().put(parameters.get(0).getPlaceholderId(),
NullLiteral.INSTANCE);
+
Assertions.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class,
+ () -> analyze(parsed.first, execute));
+ if (parameters.size() == 2) {
+
execute.getIdToPlaceholderRealExpr().put(parameters.get(0).getPlaceholderId(),
+ new StringLiteral("4"));
+
Assertions.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class,
+ () -> analyze(parsed.first, execute));
+ }
+ }
+ } finally {
+ ConnectContext.remove();
+ if (previous != null) {
+ previous.setThreadLocalInfo();
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testDistanceRangeAndMultiVectorPrepare(boolean multiVector)
throws Exception {
+ ConnectContext previous = ConnectContext.get();
+ MemoTestUtils.createConnectContext();
+ LanceExternalTable table = mockTable(new AtomicLong(42), multiVector);
+ try (MockedStatic<LanceExternalSearchTableValuedFunction> lookup =
Mockito.mockStatic(
+ LanceExternalSearchTableValuedFunction.class,
Mockito.CALLS_REAL_METHODS)) {
+ lookup.when(() ->
LanceExternalSearchTableValuedFunction.findLanceExternalTable(
+ Mockito.any(TableName.class))).thenReturn(table);
+ Map<String, String> properties = ImmutableMap.<String,
String>builder()
+ .put("table", "catalog.db.items").put("column",
"embedding").put("use_index", "false")
+ .put("distance_lower_bound",
"1").put("distance_upper_bound", "4")
+ .put("query_vector", multiVector ? "[[1,2]]" :
"[1,2]").build();
+ for (boolean prepare : new boolean[] {true, false}) {
+ if (multiVector) {
+ Exception error =
Assertions.assertThrows(org.apache.doris.common.AnalysisException.class,
+ () -> new
VectorSearchTableValuedFunction(properties, prepare));
+
Assertions.assertTrue(error.getMessage().contains("multi-vector"));
+ } else {
+ VectorSearchTableValuedFunction function = new
VectorSearchTableValuedFunction(properties, prepare);
+ Assertions.assertEquals(1,
function.getSearchRequest().getSchemaVersion());
+ Assertions.assertEquals(1.0,
function.getSearchRequest().getSearchQuery()
+ .getVectorSearch().getDistanceLowerBound());
+ Assertions.assertEquals(4.0,
function.getSearchRequest().getSearchQuery()
+ .getVectorSearch().getDistanceUpperBound());
+ Assertions.assertEquals(!prepare,
function.getSearchRequest().getSearchQuery()
+ .getVectorSearch().isSetQueryVector());
+ }
+ }
+ VectorSearchTableValuedFunction ordinary = new
VectorSearchTableValuedFunction(ImmutableMap.of(
+ "table", "catalog.db.items", "column", "embedding",
"use_index", "false"), true);
+ Assertions.assertEquals(1,
ordinary.getSearchRequest().getSchemaVersion());
+ } finally {
+ ConnectContext.remove();
+ if (previous != null) {
+ previous.setThreadLocalInfo();
+ }
+ }
+ }
+
@ParameterizedTest
@ValueSource(booleans = {true, false})
public void testPrepareAndRepeatedBindingUseFreshVectorAndSnapshot(boolean
useIndex) {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
index a141419a1ba..181abca8dc9 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java
@@ -23,7 +23,9 @@ import org.apache.doris.common.AnalysisException;
import org.apache.doris.datasource.lance.metadata.LanceTableAccess;
import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
import org.apache.doris.thrift.TVectorSearchOptions;
+import org.apache.doris.thrift.TVectorSearchParams;
+import com.google.common.collect.ImmutableMap;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
@@ -42,6 +44,51 @@ public class VectorSearchTableValuedFunctionTest {
return
VectorSearchTableValuedFunction.buildVectorSearchOptions(params, true);
}
+ @Test
+ public void testRejectInvalidDistanceBoundsBeforeMetadataAccess() {
+ for (String value : new String[] {"NaN", "Infinity", "-Infinity",
"1e100", "invalid", ""}) {
+ AnalysisException error =
Assert.assertThrows(AnalysisException.class,
+ () -> new VectorSearchTableValuedFunction(
+ Collections.singletonMap("distance_upper_bound",
value)));
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("finite FLOAT"));
+ }
+ }
+
+ @Test
+ public void testDistanceBoundsRoundTrip() throws Exception {
+
Assert.assertFalse(VectorSearchTableValuedFunction.parseDistanceBounds(Collections.emptyMap())
+ .isSetDistanceLowerBound());
+ for (Map<String, String> properties : Arrays.asList(
+ Collections.singletonMap("distance_lower_bound", "-0.5"),
+ Collections.singletonMap("distance_upper_bound", "0"),
+ ImmutableMap.of("distance_lower_bound", "0.1",
"distance_upper_bound", "1.5"))) {
+ TVectorSearchParams params =
VectorSearchTableValuedFunction.parseDistanceBounds(properties);
+ TVectorSearchParams decoded = new TVectorSearchParams();
+ new TDeserializer().deserialize(decoded, new
TSerializer().serialize(params));
+ Assert.assertEquals(params, decoded);
+
Assert.assertEquals(properties.containsKey("distance_lower_bound"),
decoded.isSetDistanceLowerBound());
+
Assert.assertEquals(properties.containsKey("distance_upper_bound"),
decoded.isSetDistanceUpperBound());
+ if (decoded.isSetDistanceLowerBound()) {
+ Assert.assertEquals((double)
Float.parseFloat(properties.get("distance_lower_bound")),
+ decoded.getDistanceLowerBound(), 0.0);
+ }
+ }
+ }
+
+ @Test
+ public void testRejectEmptyOrReversedDistanceRange() {
+ for (String upper : new String[] {"0", "-1",
"0.00000000000000000000000000000000000000000000001"}) {
+ AnalysisException error =
Assert.assertThrows(AnalysisException.class,
+ () -> new VectorSearchTableValuedFunction(
+ ImmutableMap.of("distance_lower_bound", "0",
"distance_upper_bound", upper)));
+ Assert.assertTrue(error.getMessage(),
error.getMessage().contains("must be less than"));
+ }
+ Assert.assertThrows(AnalysisException.class, () ->
VectorSearchTableValuedFunction.parseDistanceBounds(
+ ImmutableMap.of("distance_lower_bound", "1",
"distance_upper_bound", "1.00000001")));
+ Assert.assertThrows(AnalysisException.class, () ->
VectorSearchTableValuedFunction.parseDistanceBounds(
+ Collections.singletonMap("distance_lower_bound", "NaN")));
+ }
+
@Test
public void testQueryParallelismOptions() throws Exception {
Assert.assertNull(parseOptions(Collections.emptyMap()));
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 6e4fa8fd77b..6a76a159606 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -487,6 +487,9 @@ struct TVectorSearchParams {
3: optional i64 top_k
4: optional i64 offset
5: optional TVectorMetric metric
+ // FLOAT bounds: inclusive lower and exclusive upper.
+ 6: optional double distance_lower_bound
+ 7: optional double distance_upper_bound
}
enum TFtsCoverageMode {
diff --git
a/regression-test/suites/external_table_p0/lance/test_lance_vector_search_distance_range.groovy
b/regression-test/suites/external_table_p0/lance/test_lance_vector_search_distance_range.groovy
new file mode 100644
index 00000000000..f1447bf8167
--- /dev/null
+++
b/regression-test/suites/external_table_p0/lance/test_lance_vector_search_distance_range.groovy
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_lance_vector_search_distance_range", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (!"true".equalsIgnoreCase(enabled)) {
+ logger.info("Skip Lance distance range tests: Iceberg MinIO is
disabled")
+ return
+ }
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String catalogName = "test_lance_vector_search_distance_range"
+ String tableName = "${catalogName}.doris.vs_ivf_flat_f32"
+ String query = "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]"
+ def search = { boolean indexed, String options ->
+ """vector_search("table"="${tableName}", "column"="embedding",
"query_vector"="${query}",
+ "top_k"="10", "metric"="l2", "nprobes"="4",
"use_index"="${indexed}" ${options})"""
+ }
+ sql "DROP CATALOG IF EXISTS `${catalogName}`"
+ try {
+ sql """CREATE CATALOG `${catalogName}` PROPERTIES (
+ "type"="lance", "lance.catalog.type"="filesystem",
"warehouse"="s3://warehouse/lance",
+ "s3.endpoint"="http://${externalEnvIp}:${minioPort}";,
"s3.access_key"="admin",
+ "s3.secret_key"="password", "s3.region"="us-east-1",
"use_path_style"="true")"""
+ sql "SET enable_file_scanner_v2 = true"
+ // Squared L2 is 16 * (row_id - 1)^2. Full IVF_FLAT probing gives an
exact oracle.
+ for (boolean indexed : [false, true]) {
+ def cases = [
+ [', "distance_upper_bound"="16"', [1L]],
+ [', "distance_lower_bound"="16",
"distance_upper_bound"="144"', [2L, 3L]],
+ [', "distance_lower_bound"="17", "distance_upper_bound"="64"',
[]],
+ [', "distance_lower_bound"="16", "distance_upper_bound"="144",
"offset"="1"', [3L]],
+ [', "distance_lower_bound"="16", "distance_upper_bound"="144",
"filter"="row_id > 2"', [3L]],
+ [', "distance_upper_bound"="0"', []]
+ ]
+ for (def entry : cases) {
+ // Project stored columns too, exercising row-id
materialization after range selection.
+ def rows = sql "SELECT row_id, label, _distance FROM
${search(indexed, entry[0])} ORDER BY _distance"
+ assertEquals(entry[1], rows.collect { (it[0] as
Number).longValue() })
+ for (def row : rows) {
+ double expected = 16.0 * Math.pow((row[0] as
Number).longValue() - 1, 2)
+ assertEquals(expected, (row[2] as Number).doubleValue())
+ }
+ }
+ def lowerOnly = sql "SELECT row_id, _distance FROM
${search(indexed, ', \"distance_lower_bound\"=\"16\"')}"
+ assertTrue(lowerOnly.size() > 0 && lowerOnly.size() <= 10)
+ assertTrue(lowerOnly.every { (it[1] as Number).doubleValue() >=
16.0 })
+ explain {
+ sql "SELECT row_id FROM ${search(indexed, ',
\"distance_upper_bound\"=\"16\"')}"
+ contains "lanceDistanceRange=[-inf, 16.0)"
+ // Single-sided ranges must bypass the native finite-sentinel
index filter.
+ contains "lanceVectorIndexStatus=${indexed ?
'DISTANCE_RANGE_FALLBACK' : 'DISABLED'}"
+ }
+ }
+ for (String value : ["NaN", "Infinity", "1e100", "invalid"]) {
+ test {
+ sql "SELECT * FROM ${search(false, ',
\"distance_upper_bound\"=\"' + value + '\"')}"
+ exception "finite FLOAT"
+ }
+ }
+ test {
+ sql "SELECT * FROM ${search(false, ',
\"distance_lower_bound\"=\"1\", \"distance_upper_bound\"=\"1.00000001\"')}"
+ exception "must be less than"
+ }
+ } finally {
+ sql "DROP CATALOG IF EXISTS `${catalogName}`"
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
