This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/doris.git
commit d3063235507ac8e7300478b701c206eda530af0e Author: Gabriel <[email protected]> AuthorDate: Sat Oct 10 09:39:09 2026 +0800 [feature](lance) Support distance bounds in vector_search (#68815) ### What problem does this PR solve? Issue Number: Closes #66493 `vector_search()` cannot currently restrict results by distance. This adds optional `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) parameters for single-vector Radius/Range Search on Lance tables. The FE validates finite FLOAT bounds and their order, includes the range in EXPLAIN, and carries it through the external-search request. The BE applies the bounds through the existing pinned lance-c API after configuring the query vector, covering indexed and Flat Search splits, including appended data outside index coverage. The existing external-search schema version remains unchanged at 1. Results retain the existing table columns and FLOAT `_distance`, prefilter semantics, offset, and Top-K cap. Bounds can be bound through prepared-statement parameters and are revalidated on each execution. Multi-vector distance bounds are rejected. Bounded index search is limited to explicitly identified IVF_FLAT segments with both bounds present. Quantized or unknown index types and single-sided ranges fall back to exact fragment scans, reported as `DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement quantization errors and the pinned index implementation’s finite sentinels for absent bounds, at the cost of additional scanning. Unbounded queries retain their index selection; indexed IVF_FLAT recall still depends on probed partitions. Standalone BM25 is already delivered; the distributed BM25 statistics follow-up remains tracked separately in #67435. ### Release note Lance `vector_search()` supports optional lower and upper distance bounds for single-vector radius and range queries. ### Validation - BE: rebuilt the current branch and passed all 68 `LanceTableReader*` tests under ASAN, including exact SQ threshold crossing, the finite FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage. - FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`, `LancePreparedSearchTest`, `LancePreparedStatementTest`, and `LanceScanNodeTest`, including repeated bound binding, invalid parameters, all index subtypes, and indexed/flat split selection. - Native reproduction: the pinned indexed paths drop both the SQ threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat Search returns both. Added self-contained fixtures and reader coverage for these cases. - Native execution: 10 cases passed against the pinned lance-c revision, covering indexed/Flat Search, appended unindexed data, inclusive/exclusive bounds, empty results, prefiltering, and offset. - FE Checkstyle and clang-format 16 checks passed. - Added SQL regression coverage for range results, projected stored columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy syntax compilation passed; the SQL suite has not yet run against a live Doris cluster and requires External Regression CI. ### Check List (For Author) - Test - [x] Regression test added (execution pending CI) - [x] Unit Test - [x] Manual native execution test - Behavior changed: - [x] Yes: optional distance bounds on single-vector searches; existing unbounded queries retain their behavior. - Does this need documentation? - [x] Yes: bilingual 4.x Lance Catalog documentation: https://github.com/apache/doris-website/pull/4200. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --- be/src/format_v2/table/lance_reader.cpp | 33 ++++ .../auxiliary.idx | Bin 0 -> 3254 bytes .../9634c881-df64-46a5-95dc-2468928008a5/index.idx | Bin 0 -> 329 bytes .../0-388c379e-6e57-4a48-a567-603ce868bdea.txn | Bin 0 -> 205 bytes .../1-831523ce-b62e-46b0-bcd2-501920af09b8.txn | Bin 0 -> 376 bytes .../2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn | Bin 0 -> 122 bytes .../_versions/18446744073709551612.manifest | Bin 0 -> 824 bytes .../_versions/18446744073709551613.manifest | Bin 0 -> 997 bytes .../_versions/18446744073709551614.manifest | Bin 0 -> 486 bytes .../_versions/latest_version_hint.json | 1 + ...100000101111111a29c2749f892029e3946d88007.lance | Bin 0 -> 661 bytes ...000101110010011f2593f448ba30e92dc54b90567.lance | Bin 0 -> 3100 bytes .../auxiliary.idx | Bin 0 -> 3255 bytes .../76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx | Bin 0 -> 330 bytes .../0-ec982602-f60a-425e-8b31-f77179548301.txn | Bin 0 -> 205 bytes .../1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn | Bin 0 -> 378 bytes .../_versions/18446744073709551613.manifest | Bin 0 -> 1001 bytes .../_versions/18446744073709551614.manifest | Bin 0 -> 487 bytes .../_versions/latest_version_hint.json | 1 + ...000110101111101bccc4b4c19af77197f1a92b69c.lance | Bin 0 -> 3100 bytes .../auxiliary.idx | Bin 0 -> 2805 bytes .../16361222-c72d-4428-9826-beeb3bfd3667/index.idx | Bin 0 -> 327 bytes .../0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn | Bin 0 -> 205 bytes .../1-f4a6b175-3626-4561-817e-c45c857597fa.txn | Bin 0 -> 407 bytes .../_versions/18446744073709551613.manifest | Bin 0 -> 1059 bytes .../_versions/18446744073709551614.manifest | Bin 0 -> 487 bytes .../_versions/latest_version_hint.json | 1 + ...1010101000001014167dc46e9aa87a5e01e036b2a.lance | Bin 0 -> 4892 bytes .../table/lance/distance_range_fixture.md | 93 ++++++++++++ be/test/format_v2/table/lance_reader_test.cpp | 167 +++++++++++++++++++++ .../datasource/lance/source/LanceScanNode.java | 7 + .../datasource/lance/source/LanceScanPlan.java | 2 + .../datasource/lance/source/LanceScanPlanner.java | 12 ++ .../doris/nereids/parser/LogicalPlanBuilder.java | 4 +- .../VectorSearchTableValuedFunction.java | 44 +++++- .../datasource/lance/source/LanceScanNodeTest.java | 40 +++++ .../tablefunction/LancePreparedSearchTest.java | 105 +++++++++++++ .../VectorSearchTableValuedFunctionTest.java | 47 ++++++ gensrc/thrift/PlanNodes.thrift | 3 + .../test_lance_vector_search_distance_range.groovy | 82 ++++++++++ 40 files changed, 638 insertions(+), 4 deletions(-) diff --git a/be/src/format_v2/table/lance_reader.cpp b/be/src/format_v2/table/lance_reader.cpp index 73d68332f49..c445d795d94 100644 --- a/be/src/format_v2/table/lance_reader.cpp +++ b/be/src/format_v2/table/lance_reader.cpp @@ -448,6 +448,28 @@ Status LanceTableReader::_validate_external_search_request() const { static_cast<int>(query_vector.element_type)); } const bool multi_vector = query_vector.__isset.num_vectors; + const bool has_distance_range = + vector.__isset.distance_lower_bound || vector.__isset.distance_upper_bound; + if (has_distance_range && multi_vector) { + return Status::NotSupported( + "Distance bounds are not supported for Lance multi-vector search"); + } + // Validate doubles before narrowing: non-finite or out-of-range values cannot reach the C API. + const auto valid_bound = [](double bound) { + return std::isfinite(bound) && std::abs(bound) <= std::numeric_limits<float>::max(); + }; + if ((vector.__isset.distance_lower_bound && !valid_bound(vector.distance_lower_bound)) || + (vector.__isset.distance_upper_bound && !valid_bound(vector.distance_upper_bound))) { + return Status::InvalidArgument("Lance distance bounds must be finite FLOAT values"); + } + if (vector.__isset.distance_lower_bound && vector.__isset.distance_upper_bound && + static_cast<float>(vector.distance_lower_bound) >= + static_cast<float>(vector.distance_upper_bound)) { + return Status::InvalidArgument( + "distance_lower_bound must be less than distance_upper_bound at FLOAT " + "precision"); + } + // The optional count distinguishes a query matrix, including a one-row matrix. if (multi_vector && query_vector.num_vectors <= 0) { return Status::InvalidArgument( @@ -1177,6 +1199,17 @@ Status LanceTableReader::_configure_vector_search(LanceScanner* scanner, } } + if (vector.__isset.distance_lower_bound || vector.__isset.distance_upper_bound) { + const float lower = static_cast<float>(vector.distance_lower_bound); + const float upper = static_cast<float>(vector.distance_upper_bound); + // nearest() clears the range; install bounds only after setting the query vector. + if (lance_scanner_set_distance_range( + scanner, vector.__isset.distance_lower_bound ? &lower : nullptr, + vector.__isset.distance_upper_bound ? &upper : nullptr) != 0) { + return lance_error("set Lance vector distance range"); + } + } + // Exact refinement validates stored elements and gives indexed and unindexed candidates // the same row-level score before either path truncates its results. if (query.__isset.num_vectors && lance_scanner_set_refine_factor(scanner, 1) != 0) { diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/auxiliary.idx b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/auxiliary.idx new file mode 100644 index 00000000000..f6ecf98bb6c Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/auxiliary.idx differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/index.idx b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/index.idx new file mode 100644 index 00000000000..d7646e4098c Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_indices/9634c881-df64-46a5-95dc-2468928008a5/index.idx differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/0-388c379e-6e57-4a48-a567-603ce868bdea.txn b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/0-388c379e-6e57-4a48-a567-603ce868bdea.txn new file mode 100644 index 00000000000..6643a15e80a Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/0-388c379e-6e57-4a48-a567-603ce868bdea.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/1-831523ce-b62e-46b0-bcd2-501920af09b8.txn b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/1-831523ce-b62e-46b0-bcd2-501920af09b8.txn new file mode 100644 index 00000000000..949b47f39d9 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/1-831523ce-b62e-46b0-bcd2-501920af09b8.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn new file mode 100644 index 00000000000..afebdade12d Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_transactions/2-4546c7a5-c68b-47d3-8f54-1e3036ad6429.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551612.manifest b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551612.manifest new file mode 100644 index 00000000000..7ed8e3d926b Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551612.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551613.manifest b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551613.manifest new file mode 100644 index 00000000000..ba5cc92bc12 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551613.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551614.manifest b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551614.manifest new file mode 100644 index 00000000000..8f32f9b807f Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/18446744073709551614.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/_versions/latest_version_hint.json b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/latest_version_hint.json new file mode 100644 index 00000000000..f93d3984472 --- /dev/null +++ b/be/test/format_v2/table/lance/data/distance_range.lance/_versions/latest_version_hint.json @@ -0,0 +1 @@ +{"version":3} \ No newline at end of file diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/data/011111011100000101111111a29c2749f892029e3946d88007.lance b/be/test/format_v2/table/lance/data/distance_range.lance/data/011111011100000101111111a29c2749f892029e3946d88007.lance new file mode 100644 index 00000000000..52f7dbf29cc Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/data/011111011100000101111111a29c2749f892029e3946d88007.lance differ diff --git a/be/test/format_v2/table/lance/data/distance_range.lance/data/100110001000101110010011f2593f448ba30e92dc54b90567.lance b/be/test/format_v2/table/lance/data/distance_range.lance/data/100110001000101110010011f2593f448ba30e92dc54b90567.lance new file mode 100644 index 00000000000..53e1eec28d0 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range.lance/data/100110001000101110010011f2593f448ba30e92dc54b90567.lance differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/auxiliary.idx b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/auxiliary.idx new file mode 100644 index 00000000000..3ca57b8d6de Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/auxiliary.idx differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx new file mode 100644 index 00000000000..446e4f4152b Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_indices/76cb567d-f24f-4dd0-85b1-b0f486361b5e/index.idx differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/0-ec982602-f60a-425e-8b31-f77179548301.txn b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/0-ec982602-f60a-425e-8b31-f77179548301.txn new file mode 100644 index 00000000000..72e846c8998 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/0-ec982602-f60a-425e-8b31-f77179548301.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn new file mode 100644 index 00000000000..57cf3b8d551 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_transactions/1-bd4b52a8-552e-4270-b52f-3f678df92fed.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551613.manifest b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551613.manifest new file mode 100644 index 00000000000..d3526033b05 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551613.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551614.manifest b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551614.manifest new file mode 100644 index 00000000000..9aebc63bd80 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/18446744073709551614.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/latest_version_hint.json b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/latest_version_hint.json new file mode 100644 index 00000000000..218abba1699 --- /dev/null +++ b/be/test/format_v2/table/lance/data/distance_range_dot.lance/_versions/latest_version_hint.json @@ -0,0 +1 @@ +{"version":2} \ No newline at end of file diff --git a/be/test/format_v2/table/lance/data/distance_range_dot.lance/data/110000101000110101111101bccc4b4c19af77197f1a92b69c.lance b/be/test/format_v2/table/lance/data/distance_range_dot.lance/data/110000101000110101111101bccc4b4c19af77197f1a92b69c.lance new file mode 100644 index 00000000000..b72e061da89 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_dot.lance/data/110000101000110101111101bccc4b4c19af77197f1a92b69c.lance differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/auxiliary.idx b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/auxiliary.idx new file mode 100644 index 00000000000..1483b3a7c8e Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/auxiliary.idx differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/index.idx b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/index.idx new file mode 100644 index 00000000000..79948a808fe Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_indices/16361222-c72d-4428-9826-beeb3bfd3667/index.idx differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn new file mode 100644 index 00000000000..6cd36ac9668 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/0-2dfc92d6-8481-4406-8893-9fc051e167d2.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/1-f4a6b175-3626-4561-817e-c45c857597fa.txn b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/1-f4a6b175-3626-4561-817e-c45c857597fa.txn new file mode 100644 index 00000000000..b2616f9b6e6 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_transactions/1-f4a6b175-3626-4561-817e-c45c857597fa.txn differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551613.manifest b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551613.manifest new file mode 100644 index 00000000000..3a4c998b412 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551613.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551614.manifest b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551614.manifest new file mode 100644 index 00000000000..97194c3229a Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/18446744073709551614.manifest differ diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/latest_version_hint.json b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/latest_version_hint.json new file mode 100644 index 00000000000..218abba1699 --- /dev/null +++ b/be/test/format_v2/table/lance/data/distance_range_sq.lance/_versions/latest_version_hint.json @@ -0,0 +1 @@ +{"version":2} \ No newline at end of file diff --git a/be/test/format_v2/table/lance/data/distance_range_sq.lance/data/0000100001010101000001014167dc46e9aa87a5e01e036b2a.lance b/be/test/format_v2/table/lance/data/distance_range_sq.lance/data/0000100001010101000001014167dc46e9aa87a5e01e036b2a.lance new file mode 100644 index 00000000000..ac784824743 Binary files /dev/null and b/be/test/format_v2/table/lance/data/distance_range_sq.lance/data/0000100001010101000001014167dc46e9aa87a5e01e036b2a.lance differ diff --git a/be/test/format_v2/table/lance/distance_range_fixture.md b/be/test/format_v2/table/lance/distance_range_fixture.md new file mode 100644 index 00000000000..d1f3723304b --- /dev/null +++ b/be/test/format_v2/table/lance/distance_range_fixture.md @@ -0,0 +1,93 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Distance range fixture + +Synthetic single-vector data generated with `pylance==7.0.0`, using Lance data +format 2.2 and an IVF_FLAT V3 index. No external service is required to read it. +Rows 0–127 are indexed in fragment 0. Rows 128–130 were appended after indexing +in fragment 1. For query `[127, 0, 0]`, range `[1, 4)` returns rows 126 and 128, +which exercises both index coverage and the unindexed tail. + +Regenerate from the repository root: + +```python +import lance +import pyarrow as pa + +path = "be/test/format_v2/table/lance/data/distance_range.lance" +schema = pa.schema([ + pa.field("row_id", pa.int64(), nullable=False), + pa.field("embedding", pa.list_(pa.float32(), 3), nullable=False), +]) + +def batch(ids): + return pa.Table.from_pydict({ + "row_id": list(ids), + "embedding": [[float(i), 0.0, 0.0] for i in ids], + }, schema=schema) + +dataset = lance.write_dataset(batch(range(128)), path, mode="overwrite", + data_storage_version="2.2") +dataset.create_index("embedding", index_type="IVF_FLAT", num_partitions=1, + name="embedding_idx", index_file_version="V3") +lance.write_dataset(batch(range(128, 131)), path, mode="append", + data_storage_version="2.2") +``` + +Delete the previous generated dataset directory before regenerating to keep the +fixture free of obsolete files. Filenames and index UUIDs may change; tests discover +them from the manifest. + +## Range fallback fixtures + +`distance_range_sq.lance` contains vectors `[i, 0, 0]` for integers 0 through 255, +followed by row 256 with `[0.49, 0, 0]`, and an IVF_SQ V3 index. With query +`[0.51, 0, 0]`, row 256 has squared L2 distance approximately `0.0004`, inside +`[0.0001, 0.001)`. Its quantized distance is zero, so applying the lower bound +before refinement incorrectly discards it. + +`distance_range_dot.lance` contains row 0 with `[1, 0, 0]` and 127 rows with +`[0.5, 0, 0]`, and an IVF_FLAT V3 DOT index. Query `[-FLT_MAX, 0, 0]` with only +an inclusive lower bound of `FLT_MAX` must return row 0. Replacing the absent +upper bound with an exclusive `FLT_MAX` incorrectly discards it. + +These fixtures exercise fragment splits without index UUIDs, as planned by the +FE for unsafe indexed range searches, with both `use_index=true` and `false`. +They use the same schema and writer versions as the fixture above: + +```python +from pathlib import Path + +root = Path("be/test/format_v2/table/lance/data") +for name, values, index_type, metric in [ + ("distance_range_sq", [[float(i), 0.0, 0.0] for i in range(256)] + + [[0.49, 0.0, 0.0]], "IVF_SQ", "l2"), + ("distance_range_dot", [[1.0, 0.0, 0.0]] + + [[0.5, 0.0, 0.0] for _ in range(127)], "IVF_FLAT", "dot"), +]: + data = pa.Table.from_pydict({ + "row_id": list(range(len(values))), "embedding": values, + }, schema=schema) + dataset = lance.write_dataset(data, str(root / (name + ".lance")), + data_storage_version="2.2") + dataset.create_index("embedding", index_type=index_type, metric=metric, + num_partitions=1, name="embedding_idx", + index_file_version="V3") +``` diff --git a/be/test/format_v2/table/lance_reader_test.cpp b/be/test/format_v2/table/lance_reader_test.cpp index d13665f2998..aba7cfee95b 100644 --- a/be/test/format_v2/table/lance_reader_test.cpp +++ b/be/test/format_v2/table/lance_reader_test.cpp @@ -1600,6 +1600,173 @@ TEST(LanceTableReaderVectorSearchTest, RejectsInvalidQueryParallelismBeforeOpeni EXPECT_NE(std::string::npos, status.to_string().find("query_parallelism")); } +TEST(LanceTableReaderVectorSearchTest, ValidatesDistanceRangeBeforeDatasetAccess) { + TQueryGlobals globals; + RuntimeState state(globals); + const Columns columns {projected_column("_distance", TYPE_FLOAT, true)}; + for (int mode = 0; mode < 6; ++mode) { + SCOPED_TRACE(mode); + auto params = make_float32_vector_search_params({0.0F, 0.0F, 0.0F}, 2, 0); + auto& request = params.lance_scan_params.external_search_request; + auto& vector = request.search_query.vector_search; + vector.__set_distance_lower_bound(1.0); + vector.__set_distance_upper_bound(2.0); + if (mode == 0) vector.__set_distance_upper_bound(std::numeric_limits<double>::infinity()); + if (mode == 1) vector.__set_distance_lower_bound(std::numeric_limits<double>::quiet_NaN()); + if (mode == 2) vector.__set_distance_upper_bound(std::numeric_limits<double>::max()); + if (mode == 3) vector.__set_distance_upper_bound(1.0); + if (mode == 4) vector.__set_distance_upper_bound(1.00000001); + if (mode == 5) vector.query_vector.__set_num_vectors(1); + RuntimeProfile profile("invalid_distance_range"); + LanceTableReader reader; + const auto status = init_reader(&reader, columns, &state, &profile, ¶ms); + EXPECT_FALSE(status.ok()); + EXPECT_NE(std::string::npos, + status.to_string().find(mode == 5 ? "multi-vector" : "distance")); + } +} + +TEST(LanceTableReaderVectorSearchTest, AppliesDistanceBoundsAndOffset) { + const std::filesystem::path uri = "./be/test/format_v2/table/lance/data/all_types.lance"; + LanceFixtureInfo fixture; + ASSERT_TRUE(get_fixture_info(uri, &fixture).ok()); + const Columns columns {projected_column("row_id", TYPE_BIGINT, false), + projected_column("_distance", TYPE_FLOAT, true)}; + TQueryGlobals globals; + RuntimeState state(globals); + for (int mode = 0; mode < 5; ++mode) { + SCOPED_TRACE(mode); + auto params = make_float32_vector_search_params({0.0F, 0.0F, 0.0F}, 10, mode == 4 ? 1 : 0); + auto& request = params.lance_scan_params.external_search_request; + auto& vector = request.search_query.vector_search; + if (mode != 1) vector.__set_distance_lower_bound(1.0); + if (mode != 2) vector.__set_distance_upper_bound(mode == 3 ? 1.1 : 8.25); + RuntimeProfile profile("distance_range"); + LanceTableReader reader; + ASSERT_TRUE(init_reader(&reader, columns, &state, &profile, ¶ms).ok()); + ASSERT_TRUE(prepare_fixture(&reader, uri, fixture, fixture.fragment_ids).ok()); + Block block; + add_output_columns(&block, columns); + const auto rows = read_vector_search_rows(&reader, &block); + if (mode == 4) { + EXPECT_TRUE(rows.empty()); + } else { + ASSERT_FALSE(rows.empty()); + EXPECT_EQ(mode == 1 ? 1 : 2, rows[0].first); + for (const auto& [id, distance] : rows) { + if (mode != 1) EXPECT_GE(distance, 1.0F); + if (mode != 2) EXPECT_LT(distance, mode == 3 ? 1.1F : 8.25F); + } + if (mode == 0 || mode == 3) EXPECT_EQ(1U, rows.size()); + } + ASSERT_TRUE(reader.close().ok()); + } +} + +TEST(LanceTableReaderVectorSearchTest, DistanceRangeFallbackPreservesExactScores) { + const Columns columns {projected_column("row_id", TYPE_BIGINT, false), + projected_column("_distance", TYPE_FLOAT, true)}; + TQueryGlobals globals; + RuntimeState state(globals); + for (const bool dot : {false, true}) { + const std::filesystem::path uri = + dot ? "./be/test/format_v2/table/lance/data/distance_range_dot.lance" + : "./be/test/format_v2/table/lance/data/distance_range_sq.lance"; + LanceFixtureInfo fixture; + ASSERT_TRUE(get_fixture_info(uri, &fixture).ok()); + for (const bool use_index : {false, true}) { + for (const bool filtered : {false, true}) { + SCOPED_TRACE(std::to_string(dot) + "/" + std::to_string(use_index) + "/" + + std::to_string(filtered)); + const float max = std::numeric_limits<float>::max(); + auto params = make_float32_vector_search_params( + dot ? std::array<float, 3> {-max, 0.0F, 0.0F} + : std::array<float, 3> {0.51F, 0.0F, 0.0F}, + 300, 0, + filtered ? std::optional<std::string>("row_id >= 0") : std::nullopt); + auto& request = params.lance_scan_params.external_search_request; + auto& vector = request.search_query.vector_search; + vector.__set_distance_lower_bound(dot ? max : 0.0001); + if (dot) { + vector.__set_metric(TVectorMetric::DOT_PRODUCT); + } else { + vector.__set_distance_upper_bound(0.001); + } + request.vector_search_options.__set_use_index(use_index); + request.vector_search_options.__set_refine_factor(1); + request.vector_search_options.__set_query_parallelism(4); + // Range fallback splits have no index UUIDs even when use_index was requested. + // SQ quantization must not drop 0.0004, nor an absent upper bound drop FLT_MAX. + RuntimeProfile profile("distance_range_fallback"); + LanceTableReader reader; + ASSERT_TRUE(init_reader(&reader, columns, &state, &profile, ¶ms).ok()); + ASSERT_TRUE(prepare_fixture(&reader, uri, fixture, fixture.fragment_ids).ok()); + Block block; + add_output_columns(&block, columns); + const auto rows = read_vector_search_rows(&reader, &block); + ASSERT_EQ(1U, rows.size()); + EXPECT_EQ(dot ? 0 : 256, rows[0].first); + if (dot) { + EXPECT_FLOAT_EQ(max, rows[0].second); + } else { + EXPECT_NEAR(0.0004F, rows[0].second, 1e-8F); + } + ASSERT_TRUE(reader.close().ok()); + } + } + } +} + +TEST(LanceTableReaderVectorSearchTest, DistanceRangeCoversIndexedAndAppendedFragments) { + const std::filesystem::path uri = "./be/test/format_v2/table/lance/data/distance_range.lance"; + LanceFixtureInfo fixture; + ASSERT_TRUE(get_fixture_info(uri, &fixture).ok()); + ASSERT_EQ(2U, fixture.fragment_ids.size()); + std::vector<std::string> segments; + ASSERT_TRUE(get_index_segment_uuids(uri, "embedding_idx", &segments).ok()); + ASSERT_EQ(1U, segments.size()); + const Columns columns {projected_column("row_id", TYPE_BIGINT, false), + projected_column("_distance", TYPE_FLOAT, true)}; + TQueryGlobals globals; + RuntimeState state(globals); + for (const bool use_index : {false, true}) { + for (const bool filtered : {false, true}) { + std::vector<int64_t> ids; + for (size_t split = 0; split < fixture.fragment_ids.size(); ++split) { + SCOPED_TRACE(std::to_string(use_index) + "/" + std::to_string(filtered) + "/" + + std::to_string(split)); + auto params = make_float32_vector_search_params( + {127.0F, 0.0F, 0.0F}, 10, 0, + filtered ? std::optional<std::string>("row_id >= 127") : std::nullopt); + auto& request = params.lance_scan_params.external_search_request; + request.search_query.vector_search.__set_distance_lower_bound(1.0); + request.search_query.vector_search.__set_distance_upper_bound(4.0); + const bool indexed_split = use_index && split == 0; + request.vector_search_options.__set_use_index(indexed_split); + request.vector_search_options.__set_nprobes(1); + auto range = make_lance_range(uri, fixture.version, {fixture.fragment_ids[split]}); + if (indexed_split) { + range.table_format_params.lance_params.__set_index_segment_uuids(segments); + } + RuntimeProfile profile("mixed_distance_range"); + LanceTableReader reader; + ASSERT_TRUE(init_reader(&reader, columns, &state, &profile, ¶ms).ok()); + ASSERT_TRUE(prepare_range(&reader, range).ok()); + Block block; + add_output_columns(&block, columns); + for (const auto& [id, distance] : read_vector_search_rows(&reader, &block)) { + ids.push_back(id); + EXPECT_FLOAT_EQ(1.0F, distance); + } + ASSERT_TRUE(reader.close().ok()); + } + std::ranges::sort(ids); + EXPECT_EQ(filtered ? std::vector<int64_t> {128} : std::vector<int64_t>({126, 128}), + ids); + } + } +} + TEST(LanceTableReaderVectorSearchTest, QueryParallelismPreservesResults) { const std::filesystem::path uri = "./be/test/format_v2/table/lance/data/all_types.lance"; LanceFixtureInfo fixture; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java index c572234f027..4ac720cad4d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java @@ -359,6 +359,13 @@ public class LanceScanNode extends FileQueryScanNode { result.append(prefix).append("lanceQueryParallelism=") .append(externalSearchRequest.getVectorSearchOptions().getQueryParallelism()).append("\n"); } + if (vector.isSetDistanceLowerBound() || vector.isSetDistanceUpperBound()) { + result.append(prefix).append("lanceDistanceRange=[") + .append(vector.isSetDistanceLowerBound() ? vector.getDistanceLowerBound() : "-inf") + .append(", ") + .append(vector.isSetDistanceUpperBound() ? vector.getDistanceUpperBound() : "+inf") + .append(")\n"); + } result.append(prefix).append("lanceMetric=") .append(vector.isSetMetric() ? VectorSearchTableValuedFunction.metricName(vector.getMetric()) : "default") diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java index d2259c697ed..b4968f1d67f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java @@ -65,6 +65,8 @@ final class LanceScanPlan { NO_MATCH, /** The selected index has a different or unknown metric. */ METRIC_MISMATCH, + /** The pinned index search cannot safely evaluate this distance range. */ + DISTANCE_RANGE_FALLBACK, /** At least one segment has no fragment coverage metadata. */ UNKNOWN_COVERAGE, /** All segment coverage lies outside the selected snapshot's visible fragments. */ diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java index 2d97bc802bf..96925ff2399 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java @@ -35,6 +35,8 @@ import org.apache.doris.thrift.TVectorMetric; import org.apache.doris.thrift.TVectorSearchOptions; import org.apache.doris.thrift.TVectorSearchParams; +import org.lance.index.IndexType; + import java.util.ArrayList; import java.util.Collections; import java.util.LinkedHashMap; @@ -227,6 +229,16 @@ final class LanceScanPlanner { vectorIndexStatus = VectorIndexStatus.METRIC_MISMATCH; continue; } + boolean hasLower = vectorSearchParam.isSetDistanceLowerBound(); + boolean hasUpper = vectorSearchParam.isSetDistanceUpperBound(); + // Quantized indexes apply bounds before exact refinement and can discard valid rows. + // The pinned parallel IVF_FLAT path also replaces absent bounds with finite sentinels. + // Only a two-sided IVF_FLAT range is safe; unknown index types must use exact scans. + if ((hasLower || hasUpper) && (!hasLower || !hasUpper || matchingSegments.stream() + .anyMatch(segment -> segment.getIndexType() != IndexType.IVF_FLAT))) { + vectorIndexStatus = VectorIndexStatus.DISTANCE_RANGE_FALLBACK; + continue; + } Optional<LanceSplitBuilder> indexPlan = planIndexSegments( metadata, matchingSegments, visibleFragments, IndexCoveragePolicy.VECTOR_FALLBACK); if (!indexPlan.isPresent()) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java index 68cd2da1898..172d3ac2422 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java @@ -2884,7 +2884,9 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> { throw new AnalysisException("Duplicate vector_search property: " + key); } if (argument.value.constant() instanceof DorisParser.PlaceholderContext) { - if (!ImmutableSet.of("query_vector", "top_k", "offset", "filter").contains(key)) { + // Bounds change the search results, but never the prepared result schema. + if (!ImmutableSet.of("query_vector", "top_k", "offset", "filter", + "distance_lower_bound", "distance_upper_bound").contains(key)) { throw new AnalysisException("vector_search property '" + key + "' must be constant in a prepared statement"); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java index 2e9e2374b7f..d9d121924fd 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunction.java @@ -64,10 +64,13 @@ public class VectorSearchTableValuedFunction extends LanceExternalSearchTableVal private static final String NPROBES = "nprobes"; private static final String REFINE_FACTOR = "refine_factor"; private static final String EF = "ef"; + private static final String DISTANCE_LOWER_BOUND = "distance_lower_bound"; + private static final String DISTANCE_UPPER_BOUND = "distance_upper_bound"; private static final String USE_INDEX = "use_index"; private static final Set<String> PROPERTIES = ImmutableSet.of( TABLE, COLUMN, QUERY_VECTOR, TOP_K, OFFSET, METRIC, FILTER, - NPROBES, REFINE_FACTOR, EF, USE_INDEX, QUERY_PARALLELISM, VERSION, TIMESTAMP, TAG, BRANCH); + NPROBES, REFINE_FACTOR, EF, USE_INDEX, QUERY_PARALLELISM, + VERSION, TIMESTAMP, TAG, BRANCH, DISTANCE_LOWER_BOUND, DISTANCE_UPPER_BOUND); public VectorSearchTableValuedFunction(Map<String, String> properties) throws AnalysisException { @@ -82,6 +85,8 @@ public class VectorSearchTableValuedFunction extends LanceExternalSearchTableVal private static PreparedSearch prepare(Map<String, String> properties, boolean deferQueryVector) throws AnalysisException { Map<String, String> params = normalizeProperties(properties, PROPERTIES, NAME); + TVectorSearchParams vectorParams = parseDistanceBounds(params); + boolean hasDistanceRange = vectorParams.isSetDistanceLowerBound() || vectorParams.isSetDistanceUpperBound(); boolean useIndex = !params.containsKey(USE_INDEX) || parseBoolean(params.get(USE_INDEX), USE_INDEX); CommonSearch common = prepareCommon(params, NAME, @@ -89,6 +94,10 @@ public class VectorSearchTableValuedFunction extends LanceExternalSearchTableVal Field vectorField = requireSearchColumn( common.metadata().getSchema(), required(params, COLUMN, NAME), "vector"); + // Reject matrices at PREPARE as well as EXECUTE; lance-c only supports single-vector ranges. + if (hasDistanceRange && vectorField.getType() instanceof ArrowType.List) { + throw new AnalysisException("Distance bounds are not supported for Lance multi-vector search"); + } int vectorFieldId = useIndex ? requireLanceFieldId(common.metadata(), vectorField) : -1; // PREPARE needs the table schema, but the vector is supplied only at EXECUTE. @@ -100,8 +109,7 @@ public class VectorSearchTableValuedFunction extends LanceExternalSearchTableVal vectorField, required(params, QUERY_VECTOR, NAME)); } - TVectorSearchParams vectorParams = new TVectorSearchParams() - .setColumn(vectorField.getName()) + vectorParams.setColumn(vectorField.getName()) .setQueryVector(queryVector) .setTopK(common.topK()) .setOffset(common.offset()); @@ -126,6 +134,36 @@ public class VectorSearchTableValuedFunction extends LanceExternalSearchTableVal common, vectorFieldId, searchRequest, DISTANCE_COLUMN, "vector search"); } + @VisibleForTesting + static TVectorSearchParams parseDistanceBounds(Map<String, String> params) throws AnalysisException { + TVectorSearchParams vector = new TVectorSearchParams(); + if (params.containsKey(DISTANCE_LOWER_BOUND)) { + vector.setDistanceLowerBound(parseDistanceBound(params.get(DISTANCE_LOWER_BOUND), DISTANCE_LOWER_BOUND)); + } + if (params.containsKey(DISTANCE_UPPER_BOUND)) { + vector.setDistanceUpperBound(parseDistanceBound(params.get(DISTANCE_UPPER_BOUND), DISTANCE_UPPER_BOUND)); + } + // Compare after FLOAT rounding, matching the native API even for nearly equal inputs. + if (vector.isSetDistanceLowerBound() && vector.isSetDistanceUpperBound() + && vector.getDistanceLowerBound() >= vector.getDistanceUpperBound()) { + throw new AnalysisException( + "distance_lower_bound must be less than distance_upper_bound at FLOAT precision"); + } + return vector; + } + + private static float parseDistanceBound(String value, String property) throws AnalysisException { + try { + float bound = Float.parseFloat(value); + if (Float.isFinite(bound)) { + return bound; + } + } catch (NumberFormatException | NullPointerException ignored) { + // Report the SQL property rather than leaking a Java parsing exception. + } + throw new AnalysisException("'" + property + "' must be a finite FLOAT"); + } + @VisibleForTesting static TVectorSearchOptions buildVectorSearchOptions( Map<String, String> params, boolean useIndex) throws AnalysisException { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java index bd555d1dab3..32173603c8f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/source/LanceScanNodeTest.java @@ -605,6 +605,46 @@ public class LanceScanNodeTest { assertSplit(splits.get(3), 4, 8, 63); } + @Test + public void testDistanceRangesOnlyUseSafeIndexPaths() throws Exception { + for (IndexType type : IndexType.values()) { + if (type.getValue() < IndexType.VECTOR.getValue()) { + continue; + } + UUID segment = UUID.fromString("11111111-2222-3333-4444-555555555555"); + LanceTableMetadata metadata = LanceTableMetadata.createSnapshotWithIndexes( + new LanceTableAccess("s3://bucket/table.lance", Collections.emptyMap()), + 42, vectorSchema(), Arrays.asList(new LanceFragmentInfo(1, 8, 8), + new LanceFragmentInfo(2, 7, 7)), + Collections.singletonMap("vector", 9), + Collections.singletonList(new LanceIndexSegmentInfo(segment, "vector_idx", + Collections.singletonList(9), Collections.singletonList(1L), type, "L2"))); + for (int bounds = 0; bounds < 4; bounds++) { + TExternalSearchRequest request = vectorSearchRequest(5, 0); + if ((bounds & 1) != 0) { + request.getSearchQuery().getVectorSearch().setDistanceLowerBound(1); + } + if ((bounds & 2) != 0) { + request.getSearchQuery().getVectorSearch().setDistanceUpperBound(4); + } + request.setVectorSearchOptions(new TVectorSearchOptions().setUseIndex(true).setQueryParallelism(4)); + LanceScanNode node = newSearchNode(metadata, request); + List<Split> splits = node.getSplits(2); + boolean indexed = bounds == 0 || (bounds == 3 && type == IndexType.IVF_FLAT); + String status = indexed ? "USED" : "DISTANCE_RANGE_FALLBACK"; + Assert.assertTrue(type + "/" + bounds, node.getNodeExplainString("", TExplainLevel.NORMAL) + .contains("lanceVectorIndexStatus=" + status)); + Assert.assertEquals(2, splits.size()); + if (indexed) { + assertIndexSplit(splits.get(0), segment, Collections.singletonList(1L), 8, 100); + } else { + assertSplit(splits.get(0), 1, 8, 100); + } + assertSplit(splits.get(1), 2, 8, 88); + } + } + } + @Test public void testExternalSearchUsesOneSplitPerIndexSegmentAndKeepsUnindexedFragments() throws Exception { diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java index 074de75998f..7c2a2361412 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LancePreparedSearchTest.java @@ -62,9 +62,114 @@ import java.nio.charset.StandardCharsets; import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.concurrent.atomic.AtomicLong; public class LancePreparedSearchTest { + @ParameterizedTest + @ValueSource(strings = {"'distance_lower_bound'=?, 'distance_upper_bound'=?", + "'distance_lower_bound'=?", "'distance_upper_bound'=?", + "'distance_lower_bound'='-1', 'distance_upper_bound'=?", + "'distance_lower_bound'=?, 'distance_upper_bound'='4'"}) + public void testRepeatedBindingUsesFreshDistanceBounds(String bounds) { + ConnectContext previous = ConnectContext.get(); + ConnectContext context = MemoTestUtils.createConnectContext(); + LanceExternalTable table = mockTable(new AtomicLong(42)); + try (MockedStatic<LanceExternalSearchTableValuedFunction> lookup = Mockito.mockStatic( + LanceExternalSearchTableValuedFunction.class, Mockito.CALLS_REAL_METHODS)) { + lookup.when(() -> LanceExternalSearchTableValuedFunction.findLanceExternalTable( + Mockito.any(TableName.class))).thenReturn(table); + Pair<LogicalPlan, StatementContext> parsed = new NereidsParser().parseMultiple( + "select * from vector_search('table'='catalog.db.items', 'column'='embedding', " + + "'query_vector'='[1,2]', " + bounds + ")").get(0); + List<Placeholder> parameters = parsed.second.getPlaceholders(); + StatementContext prepare = MemoTestUtils.createStatementContext(context, ""); + prepare.setPrepareStage(true); + analyze(parsed.first, prepare); + for (int iteration = 0; iteration < 2; iteration++) { + StatementContext execute = MemoTestUtils.createStatementContext(context, ""); + for (int i = 0; i < parameters.size(); i++) { + execute.getIdToPlaceholderRealExpr().put(parameters.get(i).getPlaceholderId(), + new StringLiteral(Integer.toString(iteration + i))); + } + org.apache.doris.thrift.TVectorSearchParams vector = analyze(parsed.first, execute) + .getSearchRequest().getSearchQuery().getVectorSearch(); + Assertions.assertEquals(bounds.contains("distance_lower_bound"), vector.isSetDistanceLowerBound()); + Assertions.assertEquals(bounds.contains("distance_upper_bound"), vector.isSetDistanceUpperBound()); + if (vector.isSetDistanceLowerBound()) { + Assertions.assertEquals(bounds.contains("='-1'") ? -1.0 : iteration, + vector.getDistanceLowerBound()); + } + if (vector.isSetDistanceUpperBound()) { + Assertions.assertEquals(bounds.contains("='4'") ? 4.0 : iteration + parameters.size() - 1, + vector.getDistanceUpperBound()); + } + // Failed EXECUTEs must not poison the reusable unbound plan or retain old bounds. + for (String invalid : new String[] {"NaN", "Infinity", "1e39", "not-a-number"}) { + execute.getIdToPlaceholderRealExpr().put(parameters.get(0).getPlaceholderId(), + new StringLiteral(invalid)); + Assertions.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class, + () -> analyze(parsed.first, execute)); + } + execute.getIdToPlaceholderRealExpr().put(parameters.get(0).getPlaceholderId(), NullLiteral.INSTANCE); + Assertions.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class, + () -> analyze(parsed.first, execute)); + if (parameters.size() == 2) { + execute.getIdToPlaceholderRealExpr().put(parameters.get(0).getPlaceholderId(), + new StringLiteral("4")); + Assertions.assertThrows(org.apache.doris.nereids.exceptions.AnalysisException.class, + () -> analyze(parsed.first, execute)); + } + } + } finally { + ConnectContext.remove(); + if (previous != null) { + previous.setThreadLocalInfo(); + } + } + } + + @ParameterizedTest + @ValueSource(booleans = {true, false}) + public void testDistanceRangeAndMultiVectorPrepare(boolean multiVector) throws Exception { + ConnectContext previous = ConnectContext.get(); + MemoTestUtils.createConnectContext(); + LanceExternalTable table = mockTable(new AtomicLong(42), multiVector); + try (MockedStatic<LanceExternalSearchTableValuedFunction> lookup = Mockito.mockStatic( + LanceExternalSearchTableValuedFunction.class, Mockito.CALLS_REAL_METHODS)) { + lookup.when(() -> LanceExternalSearchTableValuedFunction.findLanceExternalTable( + Mockito.any(TableName.class))).thenReturn(table); + Map<String, String> properties = ImmutableMap.<String, String>builder() + .put("table", "catalog.db.items").put("column", "embedding").put("use_index", "false") + .put("distance_lower_bound", "1").put("distance_upper_bound", "4") + .put("query_vector", multiVector ? "[[1,2]]" : "[1,2]").build(); + for (boolean prepare : new boolean[] {true, false}) { + if (multiVector) { + Exception error = Assertions.assertThrows(org.apache.doris.common.AnalysisException.class, + () -> new VectorSearchTableValuedFunction(properties, prepare)); + Assertions.assertTrue(error.getMessage().contains("multi-vector")); + } else { + VectorSearchTableValuedFunction function = new VectorSearchTableValuedFunction(properties, prepare); + Assertions.assertEquals(1, function.getSearchRequest().getSchemaVersion()); + Assertions.assertEquals(1.0, function.getSearchRequest().getSearchQuery() + .getVectorSearch().getDistanceLowerBound()); + Assertions.assertEquals(4.0, function.getSearchRequest().getSearchQuery() + .getVectorSearch().getDistanceUpperBound()); + Assertions.assertEquals(!prepare, function.getSearchRequest().getSearchQuery() + .getVectorSearch().isSetQueryVector()); + } + } + VectorSearchTableValuedFunction ordinary = new VectorSearchTableValuedFunction(ImmutableMap.of( + "table", "catalog.db.items", "column", "embedding", "use_index", "false"), true); + Assertions.assertEquals(1, ordinary.getSearchRequest().getSchemaVersion()); + } finally { + ConnectContext.remove(); + if (previous != null) { + previous.setThreadLocalInfo(); + } + } + } + @ParameterizedTest @ValueSource(booleans = {true, false}) public void testPrepareAndRepeatedBindingUseFreshVectorAndSnapshot(boolean useIndex) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java index a141419a1ba..181abca8dc9 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchTableValuedFunctionTest.java @@ -23,7 +23,9 @@ import org.apache.doris.common.AnalysisException; import org.apache.doris.datasource.lance.metadata.LanceTableAccess; import org.apache.doris.datasource.lance.metadata.LanceTableMetadata; import org.apache.doris.thrift.TVectorSearchOptions; +import org.apache.doris.thrift.TVectorSearchParams; +import com.google.common.collect.ImmutableMap; import org.apache.arrow.vector.types.pojo.ArrowType; import org.apache.arrow.vector.types.pojo.Field; import org.apache.arrow.vector.types.pojo.Schema; @@ -42,6 +44,51 @@ public class VectorSearchTableValuedFunctionTest { return VectorSearchTableValuedFunction.buildVectorSearchOptions(params, true); } + @Test + public void testRejectInvalidDistanceBoundsBeforeMetadataAccess() { + for (String value : new String[] {"NaN", "Infinity", "-Infinity", "1e100", "invalid", ""}) { + AnalysisException error = Assert.assertThrows(AnalysisException.class, + () -> new VectorSearchTableValuedFunction( + Collections.singletonMap("distance_upper_bound", value))); + Assert.assertTrue(error.getMessage(), error.getMessage().contains("finite FLOAT")); + } + } + + @Test + public void testDistanceBoundsRoundTrip() throws Exception { + Assert.assertFalse(VectorSearchTableValuedFunction.parseDistanceBounds(Collections.emptyMap()) + .isSetDistanceLowerBound()); + for (Map<String, String> properties : Arrays.asList( + Collections.singletonMap("distance_lower_bound", "-0.5"), + Collections.singletonMap("distance_upper_bound", "0"), + ImmutableMap.of("distance_lower_bound", "0.1", "distance_upper_bound", "1.5"))) { + TVectorSearchParams params = VectorSearchTableValuedFunction.parseDistanceBounds(properties); + TVectorSearchParams decoded = new TVectorSearchParams(); + new TDeserializer().deserialize(decoded, new TSerializer().serialize(params)); + Assert.assertEquals(params, decoded); + Assert.assertEquals(properties.containsKey("distance_lower_bound"), decoded.isSetDistanceLowerBound()); + Assert.assertEquals(properties.containsKey("distance_upper_bound"), decoded.isSetDistanceUpperBound()); + if (decoded.isSetDistanceLowerBound()) { + Assert.assertEquals((double) Float.parseFloat(properties.get("distance_lower_bound")), + decoded.getDistanceLowerBound(), 0.0); + } + } + } + + @Test + public void testRejectEmptyOrReversedDistanceRange() { + for (String upper : new String[] {"0", "-1", "0.00000000000000000000000000000000000000000000001"}) { + AnalysisException error = Assert.assertThrows(AnalysisException.class, + () -> new VectorSearchTableValuedFunction( + ImmutableMap.of("distance_lower_bound", "0", "distance_upper_bound", upper))); + Assert.assertTrue(error.getMessage(), error.getMessage().contains("must be less than")); + } + Assert.assertThrows(AnalysisException.class, () -> VectorSearchTableValuedFunction.parseDistanceBounds( + ImmutableMap.of("distance_lower_bound", "1", "distance_upper_bound", "1.00000001"))); + Assert.assertThrows(AnalysisException.class, () -> VectorSearchTableValuedFunction.parseDistanceBounds( + Collections.singletonMap("distance_lower_bound", "NaN"))); + } + @Test public void testQueryParallelismOptions() throws Exception { Assert.assertNull(parseOptions(Collections.emptyMap())); diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift index 6e4fa8fd77b..6a76a159606 100644 --- a/gensrc/thrift/PlanNodes.thrift +++ b/gensrc/thrift/PlanNodes.thrift @@ -487,6 +487,9 @@ struct TVectorSearchParams { 3: optional i64 top_k 4: optional i64 offset 5: optional TVectorMetric metric + // FLOAT bounds: inclusive lower and exclusive upper. + 6: optional double distance_lower_bound + 7: optional double distance_upper_bound } enum TFtsCoverageMode { diff --git a/regression-test/suites/external_table_p0/lance/test_lance_vector_search_distance_range.groovy b/regression-test/suites/external_table_p0/lance/test_lance_vector_search_distance_range.groovy new file mode 100644 index 00000000000..f1447bf8167 --- /dev/null +++ b/regression-test/suites/external_table_p0/lance/test_lance_vector_search_distance_range.groovy @@ -0,0 +1,82 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_lance_vector_search_distance_range", "p0,external") { + String enabled = context.config.otherConfigs.get("enableIcebergTest") + if (!"true".equalsIgnoreCase(enabled)) { + logger.info("Skip Lance distance range tests: Iceberg MinIO is disabled") + return + } + String externalEnvIp = context.config.otherConfigs.get("externalEnvIp") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + String catalogName = "test_lance_vector_search_distance_range" + String tableName = "${catalogName}.doris.vs_ivf_flat_f32" + String query = "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]" + def search = { boolean indexed, String options -> + """vector_search("table"="${tableName}", "column"="embedding", "query_vector"="${query}", + "top_k"="10", "metric"="l2", "nprobes"="4", "use_index"="${indexed}" ${options})""" + } + sql "DROP CATALOG IF EXISTS `${catalogName}`" + try { + sql """CREATE CATALOG `${catalogName}` PROPERTIES ( + "type"="lance", "lance.catalog.type"="filesystem", "warehouse"="s3://warehouse/lance", + "s3.endpoint"="http://${externalEnvIp}:${minioPort}";, "s3.access_key"="admin", + "s3.secret_key"="password", "s3.region"="us-east-1", "use_path_style"="true")""" + sql "SET enable_file_scanner_v2 = true" + // Squared L2 is 16 * (row_id - 1)^2. Full IVF_FLAT probing gives an exact oracle. + for (boolean indexed : [false, true]) { + def cases = [ + [', "distance_upper_bound"="16"', [1L]], + [', "distance_lower_bound"="16", "distance_upper_bound"="144"', [2L, 3L]], + [', "distance_lower_bound"="17", "distance_upper_bound"="64"', []], + [', "distance_lower_bound"="16", "distance_upper_bound"="144", "offset"="1"', [3L]], + [', "distance_lower_bound"="16", "distance_upper_bound"="144", "filter"="row_id > 2"', [3L]], + [', "distance_upper_bound"="0"', []] + ] + for (def entry : cases) { + // Project stored columns too, exercising row-id materialization after range selection. + def rows = sql "SELECT row_id, label, _distance FROM ${search(indexed, entry[0])} ORDER BY _distance" + assertEquals(entry[1], rows.collect { (it[0] as Number).longValue() }) + for (def row : rows) { + double expected = 16.0 * Math.pow((row[0] as Number).longValue() - 1, 2) + assertEquals(expected, (row[2] as Number).doubleValue()) + } + } + def lowerOnly = sql "SELECT row_id, _distance FROM ${search(indexed, ', \"distance_lower_bound\"=\"16\"')}" + assertTrue(lowerOnly.size() > 0 && lowerOnly.size() <= 10) + assertTrue(lowerOnly.every { (it[1] as Number).doubleValue() >= 16.0 }) + explain { + sql "SELECT row_id FROM ${search(indexed, ', \"distance_upper_bound\"=\"16\"')}" + contains "lanceDistanceRange=[-inf, 16.0)" + // Single-sided ranges must bypass the native finite-sentinel index filter. + contains "lanceVectorIndexStatus=${indexed ? 'DISTANCE_RANGE_FALLBACK' : 'DISABLED'}" + } + } + for (String value : ["NaN", "Infinity", "1e100", "invalid"]) { + test { + sql "SELECT * FROM ${search(false, ', \"distance_upper_bound\"=\"' + value + '\"')}" + exception "finite FLOAT" + } + } + test { + sql "SELECT * FROM ${search(false, ', \"distance_lower_bound\"=\"1\", \"distance_upper_bound\"=\"1.00000001\"')}" + exception "must be less than" + } + } finally { + sql "DROP CATALOG IF EXISTS `${catalogName}`" + } +} --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
