This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new bb2a5cad5b6 [doc](lance) Document the vector element type and metric
compatibility matrix (#4082)
bb2a5cad5b6 is described below
commit bb2a5cad5b65b56f32a328019905a8db2d72c56c
Author: FANNG <[email protected]>
AuthorDate: Wed Aug 26 17:55:31 2026 +0800
[doc](lance) Document the vector element type and metric compatibility
matrix (#4082)
Documents the Lance vector index compatibility matrix requested in
[apache/doris#66495](https://github.com/apache/doris/issues/66495),
whose completion criteria
include "User documentation contains a supported algorithm compatibility
matrix". Follows up on
the [review comment on
apache/doris#66512](https://github.com/apache/doris/pull/66512#issuecomment-5324556864)
asking for the integration capability matrix to be added to this page.
Companion code PR:
[apache/doris#67039](https://github.com/apache/doris/pull/67039), which
adds
the regression coverage for the combinations documented here.
Both `docs/` and the `zh-CN` translation are updated together.
## What is added
**A vector element type x distance metric matrix** under "Supported
Vector Index Types",
describing which combinations can serve an index-backed search with the
embedded Lance version,
with footnotes for:
- the Float16 L2 index-build limitation (and that Flat Search over
Float16 is fine);
- UInt8 being hamming-only, and available only for `IVF_FLAT` and
`IVF_HNSW_FLAT`;
- Int8 having no complete index build path, plus the nullable-Int8 crash
and the upstream fix
it is waiting on.
**A "The Query Metric Must Match the Index Metric" section**, because it
is the most likely
user-visible surprise: Doris selects a vector index only when the
requested `metric` equals the
metric the index was built with, and otherwise runs Flat Search.
`EXPLAIN` reports this as
`lanceSearchIndexSegments=0`. It also notes that only one index per
vector column is considered.
## Two corrections to existing text
1. **Version binding.** The page said the BE data reader is bound to
Lance `9.1.0-beta.3` at
commit `e934cc2c`. That is the `lance-java` version the **FE** uses
(`fe/pom.xml`). The BE
reader is `lance-c v0.1.6`, which pins the Lance Rust crates at
**`7.0.0-beta.7`**, Lance
commit **`e0e977a6`**. Both are now stated separately.
The `data_storage_version` table below it stays correct:
`rust/lance-encoding/src/version.rs`
at `e0e977a6` has default `V2_1`, `Stable -> V2_1`, `Next -> V2_3`, and
`2.2` stable but not
default, which is what the table already describes.
2. **Default `metric`.** The `metric` parameter row and the paragraph
after the parameter table
both said that omitting `metric` makes Doris use the metric the index
was created with.
`LanceScanNode.metricMatches` maps an unset metric to `l2`, so on a
cosine-indexed table an
omitted `metric` plans a Flat Search instead. The documented behavior
now matches the code,
and says to pass `metric` explicitly for a non-L2 index.
---
.../lakehouse/catalogs/lance-catalog.mdx | 36 ++++++++++++++++++++--
.../lakehouse/catalogs/lance-catalog.mdx | 36 ++++++++++++++++++++--
2 files changed, 66 insertions(+), 6 deletions(-)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 972212034ce..fb8edaa8fbe 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -33,7 +33,7 @@ Lance 是面向分析和 AI 场景的列式数据格式。Doris 可以通过 Lan
## Lance 版本与兼容性
-Doris BE 数据读取器使用 `lance-c v0.1.6` 构建。该版本在 Doris 中绑定的 Lance 源码版本为
`9.1.0-beta.3`（Lance commit `e934cc2c`）。`lance-c` 和 Lance Rust crates 的版本表示
Doris 集成的读取器实现版本，与数据集中记录的 Lance `data_storage_version` 不是同一个概念。
+Doris BE 数据读取器使用 `lance-c v0.1.6` 构建，其内置的 Lance Rust crates 版本为
`7.0.0-beta.7`（Lance commit `e0e977a6`）。Doris FE 使用 `lance-java` 客户端读取
Namespace 和数据集元数据，版本为 `9.1.0-beta.3`（Lance commit `e934cc2c`）。这些版本表示 Doris
集成的读取器实现版本，与数据集中记录的 Lance `data_storage_version` 不是同一个概念。
当前读取器的文件格式兼容情况如下：
@@ -397,14 +397,14 @@ ORDER BY _distance ASC, user_id;
| `query_vector` | 是 | - | JSON 数字数组。维度必须与向量列一致，元素值必须能由向量元素类型表示。 |
| `top_k` | 否 | `10` | 跳过 `offset` 后返回的结果数，必须为正整数。 |
| `offset` | 否 | `0` | 在向量检索内部跳过的最近邻数量，必须为非负整数。`top_k + offset` 不能超过无符号 32
位整数上限。 |
-| `metric` | 否 | 匹配索引的 Metric；无索引时 `uint8` 为 `hamming`，其他支持类型为 `l2` |
距离类型：`l2`、`cosine`、`dot` 或 `hamming`。`dot_product` 是 `dot` 的别名。`uint8` 向量仅支持
`hamming`；其他当前支持的向量元素类型支持 `l2`、`cosine` 和 `dot`。 |
+| `metric` | 否 | `uint8` 为 `hamming`，其他支持类型为 `l2` | 距离类型：`l2`、`cosine`、`dot` 或
`hamming`。`dot_product` 是 `dot` 的别名。`uint8` 向量仅支持 `hamming`；其他当前支持的向量元素类型支持
`l2`、`cosine` 和 `dot`。Doris 选择向量索引时会把未设置的 `metric` 视为 `l2`，因此查询使用 `cosine` 或
`dot` 创建的索引时必须显式设置 `metric`。 |
| `filter` | 否 | - | Lance SQL 条件，在生成候选向量之前执行，即 Prefilter。 |
| `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；使用
Prefilter 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 |
| `refine_factor` | 否 | 不启用精排 | 候选集精排倍数，必须为正整数。不设置时不基于原始向量重新计算距离，量化索引返回的
`_distance` 可能是近似距离；设置为 `N` 后，Lance 先获取 `(top_k + offset) × N`
个候选，再用原始向量计算真实距离并重新排序。即使设置为 `1` 也会执行精排，因此与不设置不同。 |
| `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW
图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset)
× refine_factor)`。对非 HNSW 索引无效。 |
| `use_index` | 否 | `true` | `true` 表示将兼容的物理 Lance Index Segment 规划为索引
Split，并将未覆盖的 Fragment 保留为 Flat Search Split；如果没有可用的兼容索引元数据，则退回按 Fragment
拆分。`false` 表示每个可见 Fragment 生成一个 Split，并强制执行 Flat Search。 |
-以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，如果向量列存在兼容索引，查询使用该索引创建时配置的
Metric；不存在兼容索引或 `"use_index" = "false"` 时，`uint8` 向量使用
`hamming`，其他当前支持的向量元素类型使用 `l2`。
+以上默认值对应 Doris 当前集成的 Lance Scanner 行为。`metric` 未指定时，Doris 在选择向量索引时按 `l2`
处理，因此不会选中使用 `cosine` 或 `dot` 创建的索引。未选中索引或 `"use_index" = "false"` 时，`uint8`
向量使用 `hamming`，其他当前支持的向量元素类型使用 `l2`。
### 支持的向量索引类型
@@ -423,6 +423,36 @@ ORDER BY _distance ASC, user_id;
一个 Lance 逻辑索引可以包含多个物理 Index Segment，一个物理 Segment 也可以覆盖多个 Fragment。未被所选索引覆盖的
Fragment 不会被遗漏：Doris 会为每个这样的 Fragment 增加一个执行 Flat Search 的回退 Split。如果 FE
无法生成可用的 Index Segment 计划，则退回按 Fragment 拆分。当 `use_index=false` 时，Doris
跳过索引元数据规划，并对每个可见 Fragment 强制执行 Flat Search。Flat Search 不是一种 ANN 索引，它需要在 Lance
内直接读取并比较向量。
+### 支持的向量元素类型和距离类型
+
+索引能否用于 `vector_search()`，取决于向量元素类型以及创建索引时使用的距离类型，因为内置的 Lance
版本并未实现全部组合。下表描述使用索引检索的支持情况，其内容来自实测而非推断：元素类型 x 距离类型 x 索引类型矩阵的每个组合都在内置 Lance
版本上实际构建过，标记为可用的正是那些构建成功并能返回检索结果的组合。
+
+每个可用组合都由回归测试覆盖，其中有一项检查对全部组合生效：把检索限制到单个 IVF 分区必须改变结果。Flat Search
没有分区概念，不可能因此给出不同结果，所以这一条能区分真正走索引和静默回退。
+
+大部分组合还额外断言两件事：Doris 规划出索引分片且没有任何 Fragment
退化为无索引扫描；以及索引检索在对候选做精确距离重排后，返回的行与穷举扫描完全一致。按算法、元素类型和距离类型各取的一张代表表则是**换了一种方式**覆盖，而非更强：有提交在库中的查询结果作为基线，数据形状有闭式解时还对照可手工推导的距离值。对量化类和图类算法，这些表只**记录**它们与穷举扫描的一致性而不断言
—— 因为那种一致性来自这份冻结的 fixture，并不是算法本身的保证。所有层次都不测量召回率。
+
+| 向量元素类型 | `l2` | `cosine` | `dot` | `hamming` |
+|---|---|---|---|---|
+| `float16` | 支持，但受数值范围限制 [1] | 支持 | 支持 | 不支持 |
+| `float32` | 支持 | 支持 | 支持 | 不支持 |
+| `float64` | 支持 | 支持 | 支持 | 不支持 |
+| `uint8` | 不支持 | 不支持 | 不支持 | 仅 `IVF_FLAT` 和 `IVF_HNSW_FLAT` [2] |
+| `int8` | 仅支持 Flat Search [3] | 仅支持 Flat Search [3] | 仅支持 Flat Search [3] |
不支持 |
+
+除脚注另有说明外，表中可用的组合适用于上文列出的全部六种索引类型。
+
+1. 使用 `l2` 创建 `float16` 索引，随着数据数值增大会变得不可靠，且没有一个干净的阈值可以给出。在 16
维共线向量上实测（最大平方距离随行数增长）：2.6e5 和 1.0e6 时约一秒建成，1.7e7 时始终无法完成，而 4.2e6
时对同一份输入四次尝试只成功一次 —— 说明聚类过程的随机性也参与其中，不只取决于数据。在数值有界的向量上（例如单位量级的
embedding），六种索引类型全部可以正常构建和检索。此处并未确定成因，所以上述数字应视为观测结果而非规则。`cosine` 和 `dot`
在该范围内的每一组数据上都能成功构建，包括 `l2` 无法完成的那些规模；`dot` 还额外在同样形状放大 32 倍（坐标约
33000）的数据上构建成功，因此并非只是在 `l2` 失败的量级上未经测试。这就是已验证的范围；更接近 `float16` 上限 65504
时的行为不在其中，那时存储值本身已无
法表示。如果 `float16` 的 `l2` 索引无法构建完成，可改用 `cosine`，或使用 Flat Search —— 后者对 `float16
[...]
+2. Lance 将 `uint8` 向量视为二进制向量，因此只接受 `hamming` 距离。其 Product Quantization 和
Scalar Quantization 构建器不接受 `uint8`，所以该元素类型只能使用 `IVF_FLAT` 和 `IVF_HNSW_FLAT`。
+3. Lance 目前没有完整的 `int8` 索引构建路径，因此 `int8` 向量只能使用 Flat Search 检索。此外，在当前内置的 Lance
版本下，检索**可为 NULL** 的 `int8` 向量列会导致 BE 进程退出。在 Doris
升级到包含该修复（[lance#7498](https://github.com/lance-format/lance/pull/7498)）的
`lance-c` 版本之前，请避免检索可为 NULL 的 `int8` 向量列。
+
+Lance 在创建索引时就会拒绝的组合不会影响 Doris，例如使用 `hamming` 创建浮点类型索引，或使用 `l2`、`cosine`、`dot`
创建 `uint8` 索引。
+
+#### 查询距离类型必须与索引一致
+
+只有当查询请求的 `metric` 与创建索引时使用的距离类型一致时，Doris 才会选择该向量索引；否则 Doris 不使用索引，改为执行 Flat
Search——结果仍然正确，但需要直接扫描向量。`EXPLAIN` 可以区分两者：使用索引的计划中 `lanceSearchIndexSegments`
不为 0，未选中索引时显示 `lanceSearchIndexSegments=0`。由于未设置的 `metric` 会按 `l2` 处理，凡是不以 `l2`
创建的索引都必须显式设置 `metric`。这包括所有 `uint8` 列：它的索引只能使用 `hamming`，因此不设置 `metric`
时**必然**选不中索引、退回 Flat Search。返回的行仍然正确（flat 路径对 `uint8` 应用 `hamming`），但索引没有被使用。
+
+此外，Doris 对每个向量列只会考虑一个索引。如果同一个向量列上存在多个使用不同距离类型创建的向量索引，Doris
选中的可能并不是与查询距离类型匹配的那一个，查询会退回 Flat Search。建议每个向量列最多创建一个向量索引，使索引选择结果明确。
+
### Prefilter 和 Post-filter
TVF 的 `filter` 参数是 **Prefilter**。Doris 将该字符串传给每个搜索 Split 的 Lance Scanner，Lance
在 ANN 或 Flat Search 生成候选之前执行过滤：
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index cf60bedd4e5..20ff078fd70 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -33,7 +33,7 @@ Doris currently provides read-only access to Lance. Creating,
writing, updating,
## Lance Version and Compatibility
-The Doris BE data reader is built with `lance-c v0.1.6`. In Doris, this
version is bound to Lance `9.1.0-beta.3` at Lance commit `e934cc2c`. The
`lance-c` and Lance Rust crate versions identify the reader implementation
integrated with Doris. They are different from the Lance `data_storage_version`
recorded in a dataset.
+The Doris BE data reader is built with `lance-c v0.1.6`, which embeds the
Lance Rust crates at version `7.0.0-beta.7` (Lance commit `e0e977a6`). The
Doris FE reads Namespace and dataset metadata with the `lance-java` client at
version `9.1.0-beta.3` (Lance commit `e934cc2c`). These versions identify the
reader implementations integrated with Doris. They are different from the Lance
`data_storage_version` recorded in a dataset.
The following table describes the file-format compatibility of this reader:
@@ -397,14 +397,14 @@ Do not use the unquoted form
`lance_catalog.doris.analytics.items`; it parses as
| `query_vector` | Yes | - | JSON number array. Its dimension must match the
vector column, and each value must be representable by the vector element type.
|
| `top_k` | No | `10` | Number of results returned after skipping `offset`. It
must be a positive integer. |
| `offset` | No | `0` | Number of nearest neighbors skipped inside the vector
search. It must be a non-negative integer. `top_k + offset` must not exceed the
maximum unsigned 32-bit integer. |
-| `metric` | No | Metric of the matching index; without an index, `hamming`
for `uint8` and `l2` for other supported types | Distance metric: `l2`,
`cosine`, `dot`, or `hamming`. `dot_product` is an alias for `dot`. `uint8`
vectors support only `hamming`; the other currently supported vector element
types support `l2`, `cosine`, and `dot`. |
+| `metric` | No | `hamming` for `uint8`, `l2` for the other supported types |
Distance metric: `l2`, `cosine`, `dot`, or `hamming`. `dot_product` is an alias
for `dot`. `uint8` vectors support only `hamming`; the other currently
supported vector element types support `l2`, `cosine`, and `dot`. An unset
`metric` is treated as `l2` when Doris selects a vector index, so querying an
index built with `cosine` or `dot` requires setting `metric` explicitly. |
| `filter` | No | - | Lance SQL condition evaluated before vector candidates
are generated; that is, a Prefilter. |
| `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index
partitions to probe. It must be a positive integer. When unset, Lance starts
with one partition and can probe additional partitions when a Prefilter leaves
too few candidates. Setting it explicitly to `N` fixes both the minimum and
maximum number of probes to `N`. |
| `refine_factor` | No | Refinement disabled | Candidate refinement
multiplier. It must be a positive integer. When unset, Lance does not recompute
distances from the original vectors, so `_distance` from a quantized index may
be approximate. When set to `N`, Lance first retrieves `(top_k + offset) × N`
candidates, recomputes their exact distances from the original vectors, and
reorders them. Setting it to `1` still enables refinement and therefore differs
from leaving it unset. |
| `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained
during HNSW graph search. It must be a positive integer. If `refine_factor` is
also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It
has no effect on non-HNSW indexes. |
| `use_index` | No | `true` | When `true`, Doris plans compatible physical
Lance index segments as indexed splits and keeps uncovered Fragments as Flat
Search splits. If no usable compatible index metadata is available, Doris falls
back to Fragment splits. When `false`, Doris creates one split per visible
Fragment and forces Flat Search. |
-These defaults correspond to the Lance Scanner behavior currently integrated
with Doris. When `metric` is omitted, Doris uses the metric configured when a
compatible vector index was created. If there is no compatible index, or if
`"use_index" = "false"`, `uint8` vectors use `hamming`, while the other
currently supported vector element types use `l2`.
+These defaults correspond to the Lance Scanner behavior currently integrated
with Doris. When `metric` is omitted, Doris treats the query as `l2` while
selecting a vector index, so an index built with `cosine` or `dot` is not
selected. When no index is selected, or if `"use_index" = "false"`, `uint8`
vectors use `hamming`, while the other currently supported vector element types
use `l2`.
### Supported Vector Index Types
@@ -423,6 +423,36 @@ The embedded `lance-c v0.1.6` explicitly supports the
following Lance vector ind
A logical Lance index can contain multiple physical index segments, and one
physical segment can cover multiple Fragments. Fragments not covered by the
selected index are not omitted: Doris adds one fallback Split for each such
Fragment, which uses Flat Search. If the FE cannot construct a usable
index-segment plan, it falls back to Fragment-level splits. With
`use_index=false`, Doris skips index metadata planning and forces Flat Search
for every visible Fragment. Flat Search is not an A [...]
+### Supported Vector Element Types and Distance Metrics
+
+Whether an index can serve `vector_search()` depends on the vector element
type and the distance metric it was built with, because the embedded Lance
version does not implement every combination. The following matrix describes
index-backed search, and it is measured rather than inferred: every cell of the
element type x metric x index type matrix was built against the embedded Lance
version, and the combinations marked supported are exactly those that built and
answered a search.
+
+Every supported combination is covered by the regression suite, and one check
runs on all of them: restricting the search to a single IVF partition must
change the answer. A Flat Search has no partitions and cannot answer
differently, so this is what distinguishes a real indexed search from a silent
fallback.
+
+Most combinations are additionally asserted to plan an indexed split with no
fragment left scanned unindexed, and to return the same rows as an exhaustive
scan once their candidates are reranked with exact distances. A representative
table per algorithm, element type and metric is covered differently rather than
more: committed query results, and hand-derivable distances where the data
shape has a closed form. For the quantizing and graph algorithms those tables
record their agreement wi [...]
+
+| Vector element type | `l2` | `cosine` | `dot` | `hamming` |
+|---|---|---|---|---|
+| `float16` | Supported, with a range caveat [1] | Supported | Supported | Not
supported |
+| `float32` | Supported | Supported | Supported | Not supported |
+| `float64` | Supported | Supported | Supported | Not supported |
+| `uint8` | Not supported | Not supported | Not supported | `IVF_FLAT` and
`IVF_HNSW_FLAT` only [2] |
+| `int8` | Flat Search only [3] | Flat Search only [3] | Flat Search only [3]
| Not supported |
+
+Unless a footnote states otherwise, a supported combination applies to all six
index types listed above.
+
+1. Building a `float16` index with `l2` becomes unreliable as the magnitude of
the data grows, and there is no clean threshold to quote. Measured on
16-dimensional collinear vectors, where the largest squared distance grows with
the row count: at 2.6e5 and 1.0e6 the build finishes in about a second, at
1.7e7 it never finishes, and at 4.2e6 it finished once in four attempts over
identical input - so run-to-run variation in the clustering is part of it, not
the data alone. Over bounded vec [...]
+2. Lance treats `uint8` vectors as binary vectors, so `hamming` is the only
distance it accepts for them. Its Product Quantization and Scalar Quantization
builders reject `uint8`, which leaves `IVF_FLAT` and `IVF_HNSW_FLAT` as the
only index types available for this element type.
+3. Lance does not currently provide a complete `int8` index build path, so
`int8` vectors can only be searched with Flat Search. In addition, with the
embedded Lance version, searching a **nullable** `int8` vector column
terminates the BE process. Avoid searching nullable `int8` vector columns until
Doris upgrades to a `lance-c` release that contains the fix,
[lance#7498](https://github.com/lance-format/lance/pull/7498).
+
+Combinations that Lance rejects when the index is created never reach Doris.
Building a floating-point index with `hamming`, or a `uint8` index with `l2`,
`cosine`, or `dot`, falls into this category.
+
+#### The Query Metric Must Match the Index Metric
+
+Doris selects a vector index only when the requested `metric` equals the
metric that index was built with. Otherwise Doris does not use the index and
runs Flat Search, which still returns correct results but scans the vectors
directly. `EXPLAIN` reports the difference: an indexed plan shows a non-zero
`lanceSearchIndexSegments`, while a declined index shows
`lanceSearchIndexSegments=0`. Since an unset `metric` is treated as `l2`,
remember to set `metric` explicitly for any index not buil [...]
+
+Doris also considers only one index per vector column. If a vector column
carries several vector indexes built with different metrics, the one Doris
selects may not be the one matching the query metric, and the query then falls
back to Flat Search. Keep at most one vector index per vector column to make
index selection unambiguous.
+
### Prefilter and Post-Filter
The TVF `filter` parameter is a **Prefilter**. Doris passes the string to the
Lance Scanner for each search Split, and Lance evaluates it before ANN or Flat
Search generates candidates:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
