This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 3e0d5d9adfd [doc](lance) document multi-vector search usage and limits
(#4148)
3e0d5d9adfd is described below
commit 3e0d5d9adfd918a398b7bb4cea39808ccb4f1090
Author: Gabriel <[email protected]>
AuthorDate: Fri Sep 18 14:12:24 2026 +0800
[doc](lance) document multi-vector search usage and limits (#4148)
Document how to query a Lance multi-vector column with
`vector_search()`, including data preparation, expected results,
scoring, index usage, and parameter limits in the English and Chinese
4.x Lance Catalog guides.
Depends on https://github.com/apache/doris/pull/68028; merge the
documentation with or after that implementation. Related feature
request: https://github.com/apache/doris/issues/66494.
The guide explains supported Arrow types and nullability, row-level
L2/cosine/dot scoring, matrix dimensions, the 128-subvector limit, both
100,000-candidate budgets, mandatory multi-vector refinement, and cosine
index usage. It also corrects the metric/refinement defaults and reader
versions to match the implementation.
## Versions
- [ ] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
The Dev Lance Catalog pages were deliberately removed in
https://github.com/apache/doris-website/pull/4086. This change updates
the existing 4.x pages without restoring those pages or advertising
support in older releases. This explains the existing
missing-current-counterpart warning from the locale/version check.
## Languages
- [x] Chinese
- [x] English
## Validation
- Reviewed the complete diff against the FE parameter validation and
Lance C multi-vector scoring implementation.
- Compiled both changed MDX files with GFM/front matter support and
normalized Docusaurus explicit heading IDs.
- Changed-file link and feature checks: no errors. The external Lance
object-store documentation was checked separately.
- Verified that both locales contain identical executable examples.
Executed the data builder using pylance 11.0.0 with temporary local
storage, then checked the documented basic, offset, and prefilter
results through the patched Lance C scanner, including empty/null outer
rows.
- `git diff --check` passed.
- No full-site build or new live Doris SQL run. SEO checks reported only
existing metadata warnings; no metadata is changed.
## Docs Checklist
- [x] Checked by AI
- [x] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
---
.../lakehouse/catalogs/lance-catalog.mdx | 131 +++++++++++++++++++--
.../lakehouse/catalogs/lance-catalog.mdx | 131 +++++++++++++++++++--
2 files changed, 240 insertions(+), 22 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index da24382ebb9..6b74648a323 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -40,7 +40,7 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan
| 数据查询 | 支持列裁剪、并行扫描 Lance Fragment 和当前版本的快照一致性读取 |
| 谓词下推 | 支持将部分静态标量谓词和 Join Runtime Filter 下推到 Lance 执行 |
| 文件 TVF | 支持通过 `s3()` 和 `local()` 直接查询 Lance 数据集 |
-| 向量检索 | 使用物理 Lance Index Segment 作为并行 Split，对未覆盖的 Fragment 保留 Flat Search
Split，并由 Doris 合并全局 Top-K |
+| 向量检索 | 支持单向量和多向量列，使用物理 Lance Index Segment 作为并行 Split，对未覆盖的 Fragment 保留 Flat
Search Split，并由 Doris 合并全局 Top-K |
| 全文检索 | 通过 `full_text_search()` 使用已有的 Lance FTS 倒排索引，支持 Match OR/AND 和 Phrase
查询，并由 Doris 合并全局 Top-K |
| Lance Cache | BE 内共享索引、元数据缓存及本地磁盘数据缓存 |
| TopN 两阶段读取 | 对向量和全文检索中可延迟的输出列默认开启，由 `enable_lance_lazy_materialization` 独立控制
|
@@ -50,7 +50,7 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan
## Lance 版本与兼容性
-Doris BE 数据读取器使用 `lance-c v0.1.9` 加 Doris 补丁构建，补丁将其内置的 Lance Rust crates 升级到
`11.0.0`（commit `ab6b5bbe`）。Doris FE 使用 `org.lance:lance-core:11.0.0` 读取
Namespace 和数据集元数据，FE 与 BE 的 Lance 版本统一为 `11.0.0`。Dataset 必须同时满足 FE 元数据加载和 BE
数据读取的兼容性要求。这些实现版本与数据集中记录的 Lance `data_storage_version` 不是同一个概念。
+Doris BE 数据读取器使用 `lance-c v0.1.9` 加 Doris 补丁构建，补丁包含多向量检索支持，并将其内置的 Lance Rust
crates 升级到 `11.0.0`（commit `ab6b5bbe`）。Doris FE 使用
`org.lance:lance-core:11.0.0` 读取 Namespace 和数据集元数据，FE 与 BE 的 Lance 版本统一为
`11.0.0`。Dataset 必须同时满足 FE 元数据加载和 BE 数据读取的兼容性要求。这些实现版本与数据集中记录的 Lance
`data_storage_version` 不是同一个概念。
当前读取器的文件格式兼容情况如下：
@@ -680,22 +680,22 @@ ORDER BY _distance ASC, user_id;
| 参数 | 是否必需 | 默认值 | 说明 |
|---|---|---|---|
| `table` | 是 | - | 完整的三部分 `catalog.database.table` 名称。多级 Namespace 对应的数据库名包含
`.` 时，必须使用反引号引用数据库部分。该表必须属于 Lance Catalog，用户需要拥有该表的 `SELECT` 权限。 |
-| `column` | 是 | - | 向量列名。当前支持
`fixed_size_list<float16\|float32\|float64\|uint8\|int8>`。 |
-| `query_vector` | 是 | - | JSON 数字数组。维度必须与向量列一致，元素值必须能由向量元素类型表示。 |
+| `column` | 是 | - | 向量列名。支持单向量
`fixed_size_list<float16\|float32\|float64\|uint8\|int8>` 和多向量
`list<fixed_size_list<float16\|float32\|float64,
D>>`。详见[多向量检索](#multi-vector-search)。 |
+| `query_vector` | 是 | - | 单向量列使用 JSON 数字数组，多向量列使用非空 JSON
矩阵。每个子向量的维度必须与列一致，元素必须是向量元素类型可表示的有限数值。 |
| `top_k` | 否 | `10` | 跳过 `offset` 后返回的结果数，必须为正整数。 |
| `offset` | 否 | `0` | 在向量检索内部跳过的最近邻数量，必须为非负整数。`top_k + offset` 不能超过无符号 32
位整数上限。 |
-| `metric` | 否 | `uint8` 为 `hamming`，其他支持类型为 `l2` | 距离类型：`l2`、`cosine`、`dot` 或
`hamming`。`dot_product` 是 `dot` 的别名。`uint8` 向量仅支持 `hamming`；其他当前支持的向量元素类型支持
`l2`、`cosine` 和 `dot`。Doris 选择向量索引时会把未设置的 `metric` 视为 `l2`，因此查询使用 `cosine` 或
`dot` 创建的索引时必须显式设置 `metric`。 |
+| `metric` | 否 | `l2` | 距离类型：`l2`、`cosine`、`dot` 或 `hamming`。`dot_product` 是
`dot` 的别名。`uint8` 向量仅支持 `hamming`；其他当前支持的向量元素类型支持 `l2`、`cosine` 和 `dot`。查询
`cosine`、`dot` 或 `hamming` 索引时必须显式设置 `metric`。不支持整数多向量列及多向量 `hamming` 检索。 |
| `filter` | 否 | - | Lance SQL 条件，在生成候选向量之前执行，即 Prefilter。 |
| `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；使用
Prefilter 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 |
-| `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的
`_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N`
个候选，再用原始向量计算真实距离并重新排序。**精排会读取这些候选的原始向量数据；`N` 越大，读取和计算的候选越多，可能显著增加 I/O
并降低查询性能。** 即使设置为 `1` 也会执行精排，因此与不设置不同。 |
+| `refine_factor` | 否 | 单向量不启用；多向量为 `1` |
候选集精排倍数，必须为正整数。对于单向量检索，不设置时不基于原始向量重新计算距离，量化索引返回的 `_distance` 可能是近似距离；设置为 `N`
后，Lance 先获取 `(top_k + offset) × N` 个候选，再用原始向量计算真实距离并重新排序。**精排会读取这些候选的原始向量数据；`N`
越大，读取和计算的候选越多，可能显著增加 I/O 并降低查询性能。** 对于单向量检索，设为 `1`
会执行精排，与不设置不同。多向量检索始终使用原始向量精排，默认倍数为 `1`。 |
| `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW
图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset)
× refine_factor)`。对非 HNSW 索引无效。 |
| `use_index` | 否 | `true` | `true` 表示优先使用与向量列和距离类型兼容的 Lance 向量索引；没有可用索引时自动使用
Flat Search。`false` 表示禁用向量索引，对数据执行 Flat Search。 |
-以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，Doris 在选择向量索引时按 `l2`
处理，因此不会选中使用 `cosine` 或 `dot` 创建的索引。未选中索引或 `"use_index" = "false"` 时，`uint8`
向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。
+不设置 `metric` 时，索引检索和 Flat Search 均使用 `l2`。`uint8` 列必须显式设置 `"metric" =
"hamming"`，不支持默认的 `l2`。多向量检索还需要满足下文的[候选预算限制](#multi-vector-search)。
### 支持的向量索引类型
-当前内置 `lance-c v0.1.9` 明确支持以下 Lance 向量索引组合：
+对于单向量列，Doris 支持以下 Lance 向量索引组合。多向量的索引限制见[多向量检索](#multi-vector-search)。
| 索引类型 | 说明 | 主要查询参数 |
|---|---|---|
@@ -710,7 +710,7 @@ ORDER BY _distance ASC, user_id;
### 支持的向量元素类型和距离类型
-向量索引支持的距离类型取决于向量元素类型。请选择下表中支持的组合；不支持的组合无法使用向量索引。
+对于单向量列，向量索引支持的距离类型取决于向量元素类型。请选择下表中支持的组合；不支持的组合无法使用向量索引。
| 向量元素类型 | `l2` | `cosine` | `dot` | `hamming` |
|---|---|---|---|---|
@@ -728,14 +728,123 @@ ORDER BY _distance ASC, user_id;
#### 查询距离类型必须与索引一致
-查询指定的 `metric` 必须与索引创建时使用的距离类型一致，否则 Doris 会改用 Flat
Search。结果仍然正确，但由于需要直接扫描向量，性能通常更低。
+查询指定的 `metric` 必须与索引创建时使用的距离类型一致；否则，如果列类型支持该查询距离，Doris 会改用 Flat
Search。结果仍然正确，但由于需要直接扫描向量，性能通常更低。
-不设置 `metric` 时，Doris 在选择索引时按 `l2` 处理。因此，使用 `cosine`、`dot` 或 `hamming`
索引时必须显式设置 `metric`。例如，查询 `uint8` 向量索引时必须设置 `"metric" = "hamming"`，否则不会使用索引。
+不设置 `metric` 时，Doris 在选择索引时按 `l2` 处理。因此，使用 `cosine`、`dot` 或 `hamming`
索引时必须显式设置 `metric`。例如，查询 `uint8` 向量索引时必须设置 `"metric" = "hamming"`，否则查询使用不支持的
`l2` 距离并报错。
可以通过 `EXPLAIN` 确认是否使用索引：`lanceSearchIndexSegments` 大于 `0` 表示使用了索引，等于 `0` 表示使用
Flat Search。
Doris 对每个向量列只考虑一个向量索引。建议每个向量列最多创建一个向量索引，避免多个不同距离类型的索引导致选择不明确。
+### 多向量检索 {#multi-vector-search}
+
+多向量列在每条表记录中保存多个子向量，例如一篇文档的多个 token
embedding。一个查询矩阵产生一组按表记录排序的结果，不是批量执行多个独立向量查询，也不是同时检索多个列。
+
+#### 准备数据并执行查询
+
+使用上文的 AWS S3 Filesystem Catalog 示例，Catalog 名称为 `lance_fs_s3`，warehouse 为
`s3://my-bucket/lance`。在写入环境中安装 Lance Python SDK（`pylance`）和 `pyarrow`，配置 S3
凭证和 region，例如设置 `AWS_ACCESS_KEY_ID`、`AWS_SECRET_ACCESS_KEY` 和
`AWS_DEFAULT_REGION`。写入端需要写权限，Doris 需要同一位置的读权限。使用自定义 endpoint 或其他认证方式时，按
Catalog 配置对应的 SDK [对象存储选项](https://lance.org/guide/object_store/)。
+
+将 `documents.lance` 直接写在 warehouse 下，Doris 会将其发现为
`lance_fs_s3.default.documents`。请同时替换 Catalog 和 Python 示例中的
bucket，并使用尚不存在的数据集路径：
+
+```python
+import lance
+import pyarrow as pa
+
+vector_type = pa.list_(pa.float32(), 2)
+schema = pa.schema([
+ pa.field("id", pa.int64()),
+ pa.field("embeddings", pa.list_(
+ pa.field("item", vector_type, nullable=False)
+ )),
+])
+data = pa.Table.from_pylist([
+ {"id": 1, "embeddings": [[1.0, 0.0], [0.0, 1.0]]},
+ {"id": 2, "embeddings": [[1.0, 0.0]]},
+ {"id": 3, "embeddings": []},
+ {"id": 4, "embeddings": None},
+], schema=schema)
+lance.write_dataset(
+ data,
+ "s3://my-bucket/lance/documents.lance",
+ data_storage_version="2.1",
+)
+```
+
+外层 list 允许每条记录保存不同数量的子向量。内层的 `nullable=False` 是必需的，表示子向量本身不能为 null。使用二维 JSON
数组查询：
+
+```sql
+SELECT id, _distance
+FROM vector_search(
+ "table" = "lance_fs_s3.default.documents",
+ "column" = "embeddings",
+ "query_vector" = "[[1,0],[0,1]]",
+ "top_k" = "2",
+ "metric" = "l2",
+ "use_index" = "false"
+)
+ORDER BY _distance, id;
+```
+
+预期结果：
+
+| id | _distance |
+|---|---|
+| 1 | 0.0 |
+| 2 | 2.0 |
+
+记录 1 为每个查询子向量都提供了完全匹配。记录 2 与第一个查询子向量的距离为 `0`，与第二个的平方 L2 距离为 `2`。外层为空数组或 null
的记录不参与排名。增加 `"offset" = "1"` 并将 `"top_k" = "1"` 后，仅返回记录 2。增加 `"filter" = "id >
1"` 会先限制候选检索的记录范围，对此数据集也仅返回记录 2。
+
+#### 行级评分
+
+设查询子向量集合为 `Q`，某条记录的子向量集合为 `V`：
+
+```text
+_distance = sum(q in Q, min(v in V, distance(q, v)))
+```
+
+| `metric` | 两个子向量之间的距离 |
+|---|---|
+| `l2`（默认） | 欧氏距离的平方 |
+| `cosine` | `1 - cosine_similarity` |
+| `dot` 或 `dot_product` | `1 - dot_product`，结果可以为负数 |
+
+`_distance`
越小越好。每个查询子向量贡献一次评分，同一个存储子向量可以匹配多个查询子向量。评分取总和而非平均值，重复查询子向量会重复计算其贡献。`_distance` 的
Doris 类型始终为 `FLOAT`，包括输入为 Float64 的情况。`top_k` 和 `offset` 按表记录计数，不按子向量计数。Top-K
边界上同分记录的入选情况和顺序不保证确定；外层 `ORDER BY` 只能排序已经返回的记录。
+
+#### 类型、维度与空值
+
+| 项目 | 要求 |
+|---|---|
+| Lance 列类型 | `list<fixed_size_list<T, D>>`，其中 `D > 0`，`T` 为
`float16`、`float32` 或 `float64` |
+| Doris 类型 | Float16/Float32 对应 `ARRAY<ARRAY<FLOAT>>`，Float64 对应
`ARRAY<ARRAY<DOUBLE>>` |
+| 查询形状 | 非空 JSON 矩阵，每个内层数组必须恰好包含 `D` 个元素。即使只有一个查询子向量，也必须写成
`[[...]]`；普通单向量列仍要求使用 `[...]`。 |
+| 查询元素 | 必须是 `T` 可表示的有限数值。空矩阵、维度不一致、null、非数值元素及超出类型范围的值会被拒绝。 |
+| 存储元素 | 子向量必须声明为不可空，内部元素必须是非 null 的有限数值。评分路径遇到实际的 null 或非有限元素时会报错；仅有元素级的
nullable schema 标记是允许的。 |
+| 空记录 | 外层为 null 或 `[]` 时没有可匹配的子向量，该记录不返回。 |
+| Cosine 零范数 | 零范数向量对的 cosine
距离无定义，不能形成匹配。如果某个查询子向量在一条记录中找不到任何有效匹配，该记录被排除。因此，查询中包含零范数子向量时结果为空。 |
+| 不支持的形状和类型 | 不支持外层为 `large_list` 或 `fixed_size_list`、内层为变长
list、整数多向量元素、extension/dictionary 编码，以及多向量 `hamming` 检索。 |
+
+#### 参数限制
+
+设 `M` 为查询子向量数量，`K = top_k`，`O = offset`，`R = refine_factor`（多向量检索默认为 `1`）。
+
+| 约束 | 限制 |
+|---|---|
+| 查询子向量数量 | `1 <= M <= 128` |
+| 返回记录数与偏移量 | `K > 0`，`O >= 0` |
+| 查询候选预算 | `M × (K + O) <= 100000` |
+| 精排候选预算 | `R × (K + O) <= 100000`，且 `R > 0` |
+| 显式设置的 `nprobes`、`ef` 或 `refine_factor` | 必须为正的 32 位有符号整数，最大
`2147483647`；`refine_factor` 还受候选预算约束 |
+
+两个候选预算分别校验，包括 `use_index=false` 的情况。例如，`M=128, K=1000, O=0`
超过查询候选预算，会被拒绝；`M=4, K=1000, O=0, R=10`
同时满足两个预算。这些限制用于约束候选处理规模，不代表对内存使用量或查询耗时的保证。
+
+#### 索引使用与精排
+
+当前集成的 Lance 版本支持使用 **cosine** 距离的多向量索引。要使用已有的兼容索引，例如 cosine `IVF_FLAT` 或
`IVF_PQ`，请设置 `"metric" = "cosine"` 并保持 `use_index=true`。不设置 `metric` 时使用
`l2`；L2 和 dot 查询没有兼容索引时执行 Flat Search。Doris 不通过此函数创建索引。可以用 `SHOW INDEX`（仅
Filesystem Catalog）查看索引元数据，并通过 `EXPLAIN` 确认 `lanceSearchIndexSegments > 0`。
+
+索引检索的候选选择是近似的。多向量候选**始终使用原始向量精排**，包括不设置 `refine_factor` 或将其设为 `1`
的情况，从而使有索引与无索引的记录使用相同的行级评分。增大 `refine_factor`
会扩大候选集，增加原始向量读取和计算，但不保证完整召回。`nprobes` 控制 IVF 探测范围，`ef` 对 HNSW
索引生效；调参时应同时测量召回率与耗时。索引创建后追加的记录会通过 Flat Search 纳入检索。
+
+如需对选定记录进行穷举评分，使用 `"use_index" = "false"`。优先使用 TVF 的 `filter` 在检索前缩小候选范围；外层 SQL
`WHERE` 会丢弃候选，但不会补齐对应 split 的结果。下一节的过滤规则同时适用于单向量和多向量查询。
+
### Prefilter 和 Post-filter
Prefilter 和 Post-filter 可以在同一个查询中使用：
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index ee93c78be40..d78fd014561 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -40,7 +40,7 @@ Doris currently provides read-only access to Lance. Creating,
writing, updating,
| Data queries | Supports column pruning, parallel Lance Fragment scans, and
snapshot-consistent reads of the current version |
| Predicate pushdown | Supports pushing compatible static scalar predicates
and join Runtime Filters down to Lance |
| File TVFs | Supports querying Lance datasets directly through `s3()` and
`local()` |
-| Vector search | Uses physical Lance index segments as parallel splits, keeps
uncovered Fragments as Flat Search splits, and performs a Doris global Top-K
merge |
+| Vector search | Supports single-vector and multi-vector columns, uses
physical Lance index segments as parallel splits, keeps uncovered Fragments as
Flat Search splits, and performs a Doris global Top-K merge |
| Full-text search | Uses existing Lance FTS inverted indexes through
`full_text_search()`, supports Match OR/AND and Phrase queries, and performs a
Doris global Top-K merge |
| Lance Cache | Shares index and metadata caches and a local disk data cache
within each BE |
| Two-phase TopN read | Enabled by default for eligible output columns in
vector and full-text searches; controlled independently by
`enable_lance_lazy_materialization` |
@@ -50,7 +50,7 @@ Doris currently provides read-only access to Lance. Creating,
writing, updating,
## Lance Version and Compatibility
-The Doris BE data reader is built with `lance-c v0.1.9` plus Doris patches,
which upgrade its embedded Lance Rust crates to `11.0.0` (commit `ab6b5bbe`).
The Doris FE reads Namespace and dataset metadata with
`org.lance:lance-core:11.0.0`, aligning the FE and BE on Lance `11.0.0`. A
Dataset must be compatible with both FE metadata loading and BE data reading.
These implementation versions are different from the Lance
`data_storage_version` recorded in a dataset.
+The Doris BE data reader is built with `lance-c v0.1.9` plus Doris patches,
which add multi-vector search support and upgrade its embedded Lance Rust
crates to `11.0.0` (commit `ab6b5bbe`). The Doris FE reads Namespace and
dataset metadata with `org.lance:lance-core:11.0.0`, aligning the FE and BE on
Lance `11.0.0`. A Dataset must be compatible with both FE metadata loading and
BE data reading. These implementation versions are different from the Lance
`data_storage_version` recorded in [...]
The following table describes the file-format compatibility of this reader:
@@ -680,22 +680,22 @@ Do not use the unquoted form
`lance_catalog.doris.analytics.items`; it parses as
| Parameter | Required | Default | Description |
|---|---|---|---|
| `table` | Yes | - | Fully qualified, three-part `catalog.database.table`
name. If a multilevel Namespace maps to a database name containing `.`, quote
the database part with backticks. It must identify a table in a Lance Catalog,
and the user must have the `SELECT` privilege on the table. |
-| `column` | Yes | - | Vector column name.
`fixed_size_list<float16\|float32\|float64\|uint8\|int8>` is currently
supported. |
-| `query_vector` | Yes | - | JSON number array. Its dimension must match the
vector column, and each value must be representable by the vector element type.
|
+| `column` | Yes | - | Vector column name. Supports single-vector
`fixed_size_list<float16\|float32\|float64\|uint8\|int8>` and multi-vector
`list<fixed_size_list<float16\|float32\|float64, D>>` columns. See
[Multi-Vector Search](#multi-vector-search). |
+| `query_vector` | Yes | - | A JSON number array for a single-vector column,
or a non-empty JSON matrix for a multi-vector column. Each subvector dimension
must match the column, and each value must be finite and representable by the
element type. |
| `top_k` | No | `10` | Number of results returned after skipping `offset`. It
must be a positive integer. |
| `offset` | No | `0` | Number of nearest neighbors skipped inside the vector
search. It must be a non-negative integer. `top_k + offset` must not exceed the
maximum unsigned 32-bit integer. |
-| `metric` | No | `hamming` for `uint8`, `l2` for the other supported types |
Distance metric: `l2`, `cosine`, `dot`, or `hamming`. `dot_product` is an alias
for `dot`. `uint8` vectors support only `hamming`; the other currently
supported vector element types support `l2`, `cosine`, and `dot`. An unset
`metric` is treated as `l2` when Doris selects a vector index, so querying an
index built with `cosine` or `dot` requires setting `metric` explicitly. |
+| `metric` | No | `l2` | Distance metric: `l2`, `cosine`, `dot`, or `hamming`.
`dot_product` is an alias for `dot`. `uint8` vectors support only `hamming`;
the other currently supported vector element types support `l2`, `cosine`, and
`dot`. Set `metric` explicitly when querying a `cosine`, `dot`, or `hamming`
index. Integer multi-vector columns and multi-vector `hamming` are unsupported.
|
| `filter` | No | - | Lance SQL condition evaluated before vector candidates
are generated; that is, a Prefilter. |
| `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index
partitions to probe. It must be a positive integer. When unset, Lance starts
with one partition and can probe additional partitions when a Prefilter leaves
too few candidates. Setting it explicitly to `N` fixes both the minimum and
maximum number of probes to `N`. |
-| `refine_factor` | No | Refinement disabled | Candidate refinement
multiplier. It must be a positive integer. When unset, Lance does not recompute
distances from the original vectors, so `_distance` from a quantized index may
be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N`
candidates, recomputes their exact distances from the original vectors, and
reorders them. **Refinement reads the original vector data for these
candidates. As `N` increases, more candida [...]
+| `refine_factor` | No | Single-vector: disabled; multi-vector: `1` |
Candidate refinement multiplier. It must be a positive integer. For
single-vector search, when unset, Lance does not recompute distances from the
original vectors, so `_distance` from a quantized index may be approximate.
When set to `N`, Lance first retrieves `(top_k + offset) × N` candidates,
recomputes their exact distances from the original vectors, and reorders them.
**Refinement reads the original vector data for [...]
| `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained
during HNSW graph search. It must be a positive integer. If `refine_factor` is
also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It
has no effect on non-HNSW indexes. |
| `use_index` | No | `true` | When `true`, Doris prefers a Lance vector index
compatible with the vector column and distance metric, and automatically uses
Flat Search if no usable index is available. When `false`, Doris disables
vector indexes and performs Flat Search over the data. |
-These defaults correspond to the Lance Scanner behavior currently integrated
with Doris. When `metric` is omitted, Doris treats the query as `l2` while
selecting a vector index, so an index built with `cosine` or `dot` is not
selected. When no index is selected, or if `"use_index" = "false"`, `uint8`
vectors use `hamming`, while the other currently supported vector element types
use `l2`.
+Omitting `metric` selects `l2` for both indexed and Flat Search paths. For
`uint8` columns, explicitly set `"metric" = "hamming"`; the default `l2` is
unsupported. Multi-vector searches also enforce the [candidate
budgets](#multi-vector-search) described below.
### Supported Vector Index Types
-The embedded `lance-c v0.1.9` explicitly supports the following Lance vector
index combinations:
+For single-vector columns, Doris supports the following Lance vector index
combinations. Multi-vector index restrictions are described under [Multi-Vector
Search](#multi-vector-search).
| Index type | Description | Main query parameters |
|---|---|---|
@@ -710,7 +710,7 @@ The embedded `lance-c v0.1.9` explicitly supports the
following Lance vector ind
### Supported Vector Element Types and Distance Metrics
-The distance metrics supported by a vector index depend on the vector element
type. Choose a supported combination from the following table; unsupported
combinations cannot use a vector index.
+For single-vector columns, the distance metrics supported by a vector index
depend on the vector element type. Choose a supported combination from the
following table; unsupported combinations cannot use a vector index.
| Vector element type | `l2` | `cosine` | `dot` | `hamming` |
|---|---|---|---|---|
@@ -728,14 +728,123 @@ Unless a footnote states otherwise, "Supported" means
that the combination works
#### The Query Metric Must Match the Index Metric
-The query `metric` must match the metric used to build the index. Otherwise,
Doris uses Flat Search. The results remain correct, but performance is usually
lower because the vectors must be scanned directly.
+The query `metric` must match the metric used to build the index. Otherwise,
if the requested metric is supported for the column type, Doris uses Flat
Search. The results remain correct, but performance is usually lower because
the vectors must be scanned directly.
-When `metric` is omitted, Doris treats it as `l2` while selecting an index.
Therefore, explicitly set `metric` when using an index built with `cosine`,
`dot`, or `hamming`. For example, a `uint8` vector index requires `"metric" =
"hamming"`; otherwise, the index is not used.
+When `metric` is omitted, Doris treats it as `l2` while selecting an index.
Therefore, explicitly set `metric` when using an index built with `cosine`,
`dot`, or `hamming`. For example, a `uint8` vector index requires `"metric" =
"hamming"`; otherwise, the query uses the unsupported `l2` metric and fails.
Use `EXPLAIN` to confirm index usage. A `lanceSearchIndexSegments` value
greater than `0` indicates indexed search, while `0` indicates Flat Search.
Doris considers only one vector index per vector column. Keep at most one
vector index on each vector column to avoid ambiguous selection among indexes
built with different metrics.
+### Multi-Vector Search {#multi-vector-search}
+
+A multi-vector column stores several subvectors in each table row, for example
token embeddings for a document. A query matrix produces one ranked set of
table rows. It is not a batch of independent vector searches and does not
search several columns.
+
+#### Prepare Data and Run a Query
+
+Use the AWS S3 Filesystem Catalog example above, with catalog `lance_fs_s3`
and warehouse `s3://my-bucket/lance`. Install the Lance Python SDK (`pylance`)
and `pyarrow` in the writer environment. Configure its S3 credentials and
region, for example through `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and
`AWS_DEFAULT_REGION`. The writer needs write access, and Doris needs read
access to the same location. For custom endpoints or other authentication
methods, configure the SDK's [object- [...]
+
+Create `documents.lance` directly under the warehouse so that Doris discovers
it as `lance_fs_s3.default.documents`. Replace the bucket in both the Catalog
and Python example. Use a new dataset location:
+
+```python
+import lance
+import pyarrow as pa
+
+vector_type = pa.list_(pa.float32(), 2)
+schema = pa.schema([
+ pa.field("id", pa.int64()),
+ pa.field("embeddings", pa.list_(
+ pa.field("item", vector_type, nullable=False)
+ )),
+])
+data = pa.Table.from_pylist([
+ {"id": 1, "embeddings": [[1.0, 0.0], [0.0, 1.0]]},
+ {"id": 2, "embeddings": [[1.0, 0.0]]},
+ {"id": 3, "embeddings": []},
+ {"id": 4, "embeddings": None},
+], schema=schema)
+lance.write_dataset(
+ data,
+ "s3://my-bucket/lance/documents.lance",
+ data_storage_version="2.1",
+)
+```
+
+The outer list allows each row to have a different number of subvectors. The
inner `nullable=False` is required: a subvector itself cannot be null. Query
the table with a two-dimensional JSON array:
+
+```sql
+SELECT id, _distance
+FROM vector_search(
+ "table" = "lance_fs_s3.default.documents",
+ "column" = "embeddings",
+ "query_vector" = "[[1,0],[0,1]]",
+ "top_k" = "2",
+ "metric" = "l2",
+ "use_index" = "false"
+)
+ORDER BY _distance, id;
+```
+
+Expected result:
+
+| id | _distance |
+|---|---|
+| 1 | 0.0 |
+| 2 | 2.0 |
+
+Row 1 contains a perfect match for each query subvector. Row 2 matches the
first query subvector with distance `0` and the second with squared L2 distance
`2`. Empty and null outer lists do not rank. Adding `"offset" = "1"` and
changing `"top_k" = "1"` returns only row 2. Adding `"filter" = "id > 1"`
restricts candidate search to matching rows and also returns only row 2 for
this dataset.
+
+#### Row-Level Scoring
+
+For query subvectors `Q` and a row's stored subvectors `V`:
+
+```text
+_distance = sum(q in Q, min(v in V, distance(q, v)))
+```
+
+| `metric` | Distance between two subvectors |
+|---|---|
+| `l2` (default) | Squared Euclidean distance |
+| `cosine` | `1 - cosine_similarity` |
+| `dot` or `dot_product` | `1 - dot_product`; the value can be negative |
+
+Smaller `_distance` is better. Each query subvector contributes once, and one
stored subvector can match several query subvectors. Scores are summed, not
averaged; repeating a query subvector repeats its contribution. `_distance` has
Doris type `FLOAT`, including for Float64 inputs. `top_k` and `offset` count
table rows, not subvectors. Rows tied at the Top-K boundary have no guaranteed
membership or order; an outer `ORDER BY` only orders the returned rows.
+
+#### Types, Dimensions, and Null Values
+
+| Item | Requirement |
+|---|---|
+| Lance column | `list<fixed_size_list<T, D>>`, where `D > 0` and `T` is
`float16`, `float32`, or `float64` |
+| Doris type | `ARRAY<ARRAY<FLOAT>>` for Float16/Float32;
`ARRAY<ARRAY<DOUBLE>>` for Float64 |
+| Query shape | A non-empty JSON matrix; every inner array has exactly `D`
elements. Even a single query subvector must be written as `[[...]]`. Ordinary
single-vector columns still require `[...]`. |
+| Query values | Finite numbers representable by `T`. Empty matrices, unequal
dimensions, nulls, nonnumeric elements, and out-of-range values are rejected. |
+| Stored values | Subvectors must be declared non-nullable and contain finite,
non-null elements. The scoring path rejects actual null or non-finite elements
when encountered; a nullable *element* schema flag alone is allowed. |
+| Empty or null rows | A null outer list or `[]` has no matching subvector and
is excluded from results. |
+| Cosine zero norms | A zero-norm pair has undefined cosine distance and
cannot match. A row is excluded if any query subvector has no defined match. A
zero-norm query subvector therefore produces no results. |
+| Unsupported shapes and types | Outer `large_list` or `fixed_size_list`,
variable-length inner lists, integer multi-vector elements,
extension/dictionary encodings, and `hamming` multi-vector searches are not
supported. |
+
+#### Parameter Limits
+
+Let `M` be the number of query subvectors, `K = top_k`, `O = offset`, and `R =
refine_factor` (default `1` for multi-vector search).
+
+| Constraint | Limit |
+|---|---|
+| Query subvector count | `1 <= M <= 128` |
+| Result count and offset | `K > 0`, `O >= 0` |
+| Query candidate budget | `M × (K + O) <= 100000` |
+| Refinement candidate budget | `R × (K + O) <= 100000`, with `R > 0` |
+| Explicit `nprobes`, `ef`, or `refine_factor` | Positive 32-bit signed
integer, at most `2147483647`; the candidate budgets further constrain
`refine_factor` |
+
+Both candidate budgets apply independently, including with `use_index=false`.
For example, `M=128, K=1000, O=0` is rejected because the query candidate
budget is exceeded. `M=4, K=1000, O=0, R=10` satisfies both budgets. These
limits bound candidate work; they do not guarantee a particular memory usage or
query latency.
+
+#### Index Usage and Refinement
+
+The integrated Lance version supports multi-vector indexes with the **cosine**
metric. To use an existing compatible index, such as a cosine `IVF_FLAT` or
`IVF_PQ` index, set `"metric" = "cosine"` and leave `use_index=true`. Omitting
`metric` selects `l2`; L2 and dot searches use Flat Search when no compatible
index exists. Doris does not create indexes through this function. Use `SHOW
INDEX` (Filesystem Catalog only) to inspect index metadata and `EXPLAIN` to
check that `lanceSearchInde [...]
+
+With indexed search, candidate selection is approximate. Multi-vector
candidates are **always refined against the original vectors**, including when
`refine_factor` is omitted or set to `1`, so indexed and unindexed rows use the
same row-level score. Increasing `refine_factor` expands the candidate pool,
with additional vector reads and computation; it does not guarantee exhaustive
recall. `nprobes` controls IVF probing and `ef` applies to HNSW indexes. Tune
these only after measuring re [...]
+
+For exhaustive scoring of the selected rows, use `"use_index" = "false"`.
Prefer the TVF `filter` to restrict candidates before search; an outer SQL
`WHERE` can discard candidates without refilling a split's results. The
filtering rules in the next section apply to both single-vector and
multi-vector searches.
+
### Prefilter and Post-Filter
Prefilter and Post-filter can be used in the same query:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
