This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new da6140834c38 [docs](lance) Explain approximation modes and ACORN
selection (#4554)
da6140834c38 is described below
commit da6140834c387900629d9f34d835806475ef0dda
Author: Gabriel <[email protected]>
AuthorDate: Sat Oct 31 16:50:08 2026 +0800
[docs](lance) Explain approximation modes and ACORN selection (#4554)
## What is changed
Document the `approx_mode` property added by
https://github.com/apache/doris/pull/69852 in the English and Chinese
4.x Lance Catalog pages. This is the companion website documentation for
that feature; merge it together with or after the implementation.
The parameter reference and new guidance cover:
- Accepted values and default omission; index-specific differences
between `fast`, `normal`, and `accurate`.
- A workload selection table and tuning order using exact ground truth,
recall, latency, and original-vector refinement.
- ACORN prerequisites, a SQL example, per-HNSW-subindex filtering
thresholds, fallback behavior, and the limits of EXPLAIN.
- The coupled ACORN and RabitQ scoring effects on compatible HNSW/RabitQ
indexes; HNSW_FLAT measurements do not establish the combined index's
recall.
- Lookup-table precision versus stored encoding bits, approximate versus
exact results, and FE/BE upgrade compatibility.
The compatibility note now also describes FE rejection of explicit
FAST/ACCURATE when the initialized query backend policy contains a
marked old smooth-upgrade source, the omitted/NORMAL exception, the
error's recovery action, and the limit for unmarked older builds.
The detailed native validation measurements are recorded in the
implementation PR. The temporary guide in the Doris source repository is
being removed in favor of this website documentation.
## Validation and scope
- Both changed pages compiled successfully with the Docusaurus MDX
processor.
- Ran the documentation governance checks and compared them with the
same checks on the unchanged base content. No new warnings or errors;
the only new informational entries report the public implementation-PR
links.
- Reviewed the complete diff and Chinese/English parity. Existing URLs,
heading anchors, navigation entries, front matter, and unrelated content
are preserved. No new content issues found.
- The existing sidebar/SEO warnings are unchanged. The version-sync
warning is intentionally explained here: the implementation targets
**branch-4.2**, so only the **4.x** pages are updated. The option has
not been added to the Doris master/development implementation; copying
the guidance to development-version docs would claim support before that
implementation exists. Other locales retain their existing content.
## Versions
- [ ] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI: full self-review completed, including content
correctness, locale/version scope, links, anchors, metadata, and example
consistency.
- [x] Test Cases Built: both changed MDX pages compiled; governance
delta checked against the base; mode behavior was exercised in the
companion implementation's native validation.
- [x] Updated required version and language counterparts, or explained
why not.
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync: both English and Chinese are updated.
---
.../lakehouse/catalogs/lance-catalog.mdx | 77 ++++++++++++++++++++++
.../lakehouse/catalogs/lance-catalog.mdx | 77 ++++++++++++++++++++++
2 files changed, 154 insertions(+)
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 3457a7e1b371..ad3e31aaeb32 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -780,11 +780,88 @@ ORDER BY _distance ASC, user_id;
| `nprobes` | 否 | 最少 `1`，不限制最大值 | IVF 索引探测的分区数量，必须为正整数。不设置时从 1 个分区开始；设置
`prefilter=true` 且候选不足时，Lance 可以继续探测更多分区。显式设置为 `N` 时，最少和最多探测数都会固定为 `N`。 |
| `refine_factor` | 否 | 单向量不启用；多向量为 `1` |
候选集精排倍数，必须为正整数。对于单向量检索，不设置时不基于原始向量重新计算距离，量化索引返回的 `_distance` 可能是近似距离；设置为 `N`
后，Lance 先获取 `(top_k + offset) × N` 个候选，再用原始向量计算真实距离并重新排序。**精排会读取这些候选的原始向量数据；`N`
越大，读取和计算的候选越多，可能显著增加 I/O 并降低查询性能。** 对于单向量检索，设为 `1`
会执行精排，与不设置不同。多向量检索始终使用原始向量精排，默认倍数为 `1`。 |
| `ef` | 否 | `floor(1.5 × (top_k + offset))` | HNSW
图索引搜索时保留的候选宽度，必须为正整数。如果同时设置了 `refine_factor`，默认值为 `floor(1.5 × (top_k + offset)
× refine_factor)`。对非 HNSW 索引无效。 |
+| `approx_mode` | 否 | Lance 默认策略，当前为 `normal` | 近似检索策略，取字符串 `fast`、`normal` 或
`accurate`，忽略大小写和首尾空格。其他值（包括数字字符串）会被拒绝。适用于单向量和多向量查询。见[选择近似检索模式](#vector-approx-mode)。
|
| `use_index` | 否 | `true` | `true` 表示优先使用与向量列和距离类型兼容的 Lance 向量索引；没有可用索引时自动使用
Flat Search。`false` 表示禁用向量索引，对数据执行 Flat Search。 |
| `version`、`timestamp`、`tag`、`branch` | 否 | - | 检索历史版本、某个时间点的版本、tag 或
branch，而不是 main 的最新版本。见[检索历史版本或
branch](#search-a-historical-version-or-branch)。 |
不设置 `metric` 时，索引检索和 Flat Search 均使用 `l2`。`uint8` 列必须显式设置 `"metric" =
"hamming"`，不支持默认的 `l2`。多向量检索还需要满足下文的[候选预算限制](#multi-vector-search)。
+### 选择近似检索模式 {#vector-approx-mode}
+
+建议先使用 `normal` 建立基线。`approx_mode` 调整的是原生检索算法的策略，实际效果取决于所选索引和执行路径。它不改变
`nprobes`、`ef`、`refine_factor`、已存储的索引或 Split 规划。省略该参数时保留 Lance 的默认策略，当前为
`normal`。
+
+:::note
+该参数需要包含[近似检索模式实现](https://github.com/apache/doris/pull/69852)的 Doris
构建。下文描述的是该实现采用的原生 Lance 版本。较早的 FE 会拒绝该属性，较早的 BE 则会忽略新增的可选字段。在平滑升级期间，只要查询的候选 BE
中仍有被标记为旧升级源的节点，FE 就会拒绝显式指定的 `fast` 或 `accurate`，避免不同 Split 混用不同策略。错误信息会指出相关 BE
和请求的模式。请完成这些 BE 的升级，或省略该参数/使用 `normal`；省略参数和显式 `normal`
都保留原生默认策略，仍可兼容这些升级源节点。依赖非默认模式前，应升级所有参与查询的 FE 和 BE；该检查无法识别未被标记为平滑升级源的任意旧构建。
+:::
+
+#### 三种模式的区别
+
+| 检索路径 | `fast` | `normal` | `accurate` |
+|---|---|---|---|
+| RabitQ raw-query 距离估算 | 即使索引为多 bit 编码，也使用 one-bit 分数，跳过额外编码位的计算及其对应的距离下界筛选。
| 使用标准的 8-bit 量化距离查找表，并在可用时利用额外编码位进一步计算。 | 使用精度更高的 16-bit
距离查找表，并在可用时利用额外编码位进一步计算。 |
+| HNSW 前置过滤检索 | 满足下文运行时条件时启用 ACORN 遍历。 | 使用常规过滤图遍历。 | 图遍历策略与 `normal`
相同，量化距离计算仍可能不同。 |
+| IVF_FLAT 或全量 Flat Search | 不引入模式特有的量化变化。 | 距离计算策略相同。 | 距离计算策略相同。 |
+
+这里的 8-bit、16-bit 指**查询时距离查找表的精度**，不是索引中每个维度存储的位数。例如，同一个 5-bit RabitQ 索引，在
raw-query 路径下，`fast` 使用 one-bit 分数，另外两种模式可以利用额外的 4 bit 编码。其他 RabitQ 路径（包括
residual-query 距离估算）的行为可能不同。模式说明不代表扩大了支持的向量类型或距离类型组合。
+
+`accurate` 仍是近似检索，不保证精确 Top-K、遍历全部候选，或在量化索引上直接返回原始向量的
`_distance`。`refine_factor`
控制使用原始向量精排；增大它可以改善分数和召回率，但无法找回所有在前面阶段遗漏的候选。多向量查询保留原有的强制原始向量精排行为。
+
+#### 什么时候使用哪种模式
+
+| 场景 | 建议起点 | 需要验证的内容 |
+|---|---|---|
+| 常规负载，或尚未明确瓶颈 | `normal` | 先建立召回率和延迟基线，再调整策略。 |
+| HNSW_FLAT 有实际生效的前置过滤，并且关注延迟 | 对比 `fast` 与 `normal` | ACORN
可以减少无效距离计算，但不保证加速；同时检查召回率、返回数量及延迟中位数和 p95。 |
+| 多 bit RabitQ，业务重视召回率 | `normal` | `fast`
会舍弃部分有用的编码信息，可能显著降低召回率；只有代表性查询仍达到质量目标时才使用。 |
+| RabitQ 未做原始向量精排，关注近似分数误差 | 对比 `accurate` 与 `normal` |
对照原始向量距离检查分数误差、排序质量及额外延迟；估算更精细不保证召回率更高。 |
+| RabitQ 已使用原始向量精排 | `normal` | `accurate` 对最终分数的额外收益可能较小；候选选择仍可能不同，需要单独验证召回率。
|
+| IVF_FLAT，或没有有效前置过滤的 HNSW_FLAT | 保持 `normal` | 当前实现中，切换模式没有对应的量化或 ACORN
收益；IVF_FLAT 的召回率仍取决于探测的分区。 |
+
+对于兼容的 **HNSW 与 RabitQ 组合索引**，`fast` 可能同时启用 ACORN 遍历，并在适用的 raw-query 路径中简化
RabitQ 距离估算。这两种效果由同一个参数控制，不能根据 HNSW_FLAT 的测试结果推断它的召回率，必须单独验证组合索引。`normal` 与
`accurate` 的 HNSW 遍历策略相同，但 RabitQ 距离计算仍可能不同。
+
+建议按以下顺序调优：
+
+1. 固定相同的数据集快照、查询集、metric、过滤条件和索引覆盖范围。使用 `use_index=false`，对相同的合格行计算精确 Top-K
基准；资格过滤条件放在 TVF 的 `filter` 中，并设置 `prefilter=true`。
+2. 使用 `normal` 建立基线，测量 Recall@K、返回数量、相关场景下的分数误差，以及延迟中位数和 p95。
+3. 召回率不足时调整 IVF 的 `nprobes`、HNSW 的 `ef`；需要原始向量分数或更多精排候选时配置
`refine_factor`，同时考虑额外的向量读取和计算开销。
+4. 固定其他参数，对过滤 HNSW 测试 `fast`，或对 RabitQ 的估算精度测试
`accurate`。只有达到质量目标且实测收益值得时才采用，并分别测量冷、热缓存下的效果。
+
+#### 为带过滤的 HNSW 启用 ACORN
+
+实际选中的兼容索引必须包含 HNSW 图。设置 `approx_mode="fast"`，并将过滤条件放入 TVF，设置
`prefilter="true"`。外层 Doris `WHERE` 不提供这种前置过滤，仅设置 `use_index=true` 也不能强制选中 HNSW
索引。
+
+沿用语法示例中的四维 `embedding`、`user_id` 和字符串 `category` 列，提前创建兼容的 L2 HNSW 索引，并确保用户拥有
`SELECT` 权限：
+
+```sql
+SELECT user_id, _distance
+FROM vector_search(
+ "table" = "lance_catalog.default.items",
+ "column" = "embedding",
+ "query_vector" = "[0.1,0.2,0.3,0.4]",
+ "metric" = "l2",
+ "top_k" = "10",
+ "filter" = "category = 'book'",
+ "prefilter" = "true",
+ "use_index" = "true",
+ "approx_mode" = "fast",
+ "ef" = "64"
+)
+ORDER BY _distance, user_id;
+```
+
+该查询返回最多 10 条满足条件、按距离排序的记录，实际 ID 和召回率取决于数据及 ANN 检索。对该查询执行 `EXPLAIN` 会显示
`lanceApproxMode=FAST`，表示指定了该策略，不能证明每个子索引最终都执行了 ACORN。
+
+当前原生实现会在每个 HNSW 子索引（一个 IVF 分区）内，根据通过前置过滤的节点数量分别选择路径：
+
+| 运行时条件 | 执行路径 |
+|---|---|
+| 没有有效前置过滤，或所有节点都通过 | 普通 HNSW 遍历。 |
+| 通过的节点少于约 10% | 对剩余节点进行 Flat 距离计算。 |
+| 部分节点通过，且未触发上述稀疏过滤规则 | `fast` 使用 ACORN，`normal` 和 `accurate` 使用常规过滤图遍历。 |
+| ACORN 找到的必要候选数量不足，且查询没有距离边界 | 重新执行常规过滤图遍历。 |
+
+该阈值针对每个子索引判断，不是按整张表的过滤比例判断。此接口没有独立的强制 ACORN 属性。在 Doris
中，距离范围的正确性规则还可能在进入原生图检索前就选择 Flat Search，见[半径与范围检索](#半径与范围检索)。
+
### 半径与范围检索
设置 `distance_upper_bound` 可执行半径检索，同时设置上下界可查询区间 `[lower,
upper)`，未设置的一侧不限制距离。距离含义由所选 metric 决定：`l2` 是**欧氏距离的平方**，因此几何半径 `r` 对应的距离上界为 `r
* r`。允许负数边界，适用于 `dot` 等度量。即使存储向量为 Float64，边界也使用 `FLOAT` 精度；非有限值、溢出、相等或反向区间都会被拒绝。
diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
index 3a952fff3a95..e1071ef3f08a 100644
--- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
+++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx
@@ -780,11 +780,88 @@ Do not use the unquoted form
`lance_catalog.doris.analytics.items`; it parses as
| `nprobes` | No | Minimum `1`, with no maximum | Number of IVF index
partitions to probe. It must be a positive integer. When unset, Lance starts
with one partition and can probe additional partitions when `prefilter=true`
leaves too few candidates. Setting it explicitly to `N` fixes both the minimum
and maximum number of probes to `N`. |
| `refine_factor` | No | Single-vector: disabled; multi-vector: `1` |
Candidate refinement multiplier. It must be a positive integer. For
single-vector search, when unset, Lance does not recompute distances from the
original vectors, so `_distance` from a quantized index may be approximate.
When set to `N`, Lance first retrieves `(top_k + offset) × N` candidates,
recomputes their exact distances from the original vectors, and reorders them.
**Refinement reads the original vector data for [...]
| `ef` | No | `floor(1.5 × (top_k + offset))` | Candidate width retained
during HNSW graph search. It must be a positive integer. If `refine_factor` is
also set, the default is `floor(1.5 × (top_k + offset) × refine_factor)`. It
has no effect on non-HNSW indexes. |
+| `approx_mode` | No | Native default, currently `normal` | String policy for
approximate search: `fast`, `normal`, or `accurate`, ignoring case and
surrounding whitespace. Other values, including numeric strings, are rejected.
Applies to single-vector and multi-vector queries. See [Choose an Approximation
Mode](#vector-approx-mode). |
| `use_index` | No | `true` | When `true`, Doris prefers a Lance vector index
compatible with the vector column and distance metric, and automatically uses
Flat Search if no usable index is available. When `false`, Doris disables
vector indexes and performs Flat Search over the data. |
| `version`, `timestamp`, `tag`, `branch` | No | - | Search a historical
version, the version at a time, a tag, or a branch instead of the latest
version of `main`. See [Search a Historical Version or
Branch](#search-a-historical-version-or-branch). |
Omitting `metric` selects `l2` for both indexed and Flat Search paths. For
`uint8` columns, explicitly set `"metric" = "hamming"`; the default `l2` is
unsupported. Multi-vector searches also enforce the [candidate
budgets](#multi-vector-search) described below.
+### Choose an Approximation Mode {#vector-approx-mode}
+
+Use `normal` as the initial baseline. `approx_mode` changes the native search
algorithm's policy; its effect depends on the selected index and execution
path. It does not change `nprobes`, `ef`, `refine_factor`, the stored index, or
the split plan. Omitting it preserves Lance's default, currently `normal`.
+
+:::note
+This option requires a Doris build containing [the approximation-mode
implementation](https://github.com/apache/doris/pull/69852). The details below
describe the native Lance revision used by that implementation. Earlier FEs
reject the property, and earlier BEs ignore the optional field. During a smooth
upgrade, FE rejects explicit `fast` or `accurate` if any backend in the query's
candidate pool is marked as an old upgrade source, preventing different splits
from using different policie [...]
+:::
+
+#### Differences Between Modes
+
+| Search path | `fast` | `normal` | `accurate` |
+|---|---|---|---|
+| RabitQ raw-query distance estimation | Uses one-bit scores even for a
multi-bit index, skipping extension-bit computation and its lower-bound gate. |
Uses the standard 8-bit quantized distance lookup table and extension-bit
computation when available. | Uses a higher-precision 16-bit distance lookup
table and extension-bit computation when available. |
+| HNSW with prefiltering | Enables ACORN traversal when the runtime conditions
below permit it. | Uses regular filtered graph traversal. | Uses the same
traversal policy as `normal`; quantized distance computation can still differ. |
+| IVF_FLAT or exhaustive Flat Search | No mode-specific quantization change. |
Same scoring policy. | Same scoring policy. |
+
+The 8-bit and 16-bit values describe the **query-time distance lookup table**,
not the number of bits stored per dimension in the index. For example, on a
5-bit RabitQ index, `fast` uses one-bit scores on the raw-query path; the other
modes can use the additional four bits. Other RabitQ paths, including
residual-query scoring, can behave differently. A mode description does not
expand the supported vector-type or metric combinations.
+
+`accurate` is still approximate search. It does not guarantee exact Top-K,
exhaustive candidate evaluation, or original-vector `_distance` on a quantized
index. `refine_factor` controls reranking with original vectors; increasing it
can improve scores and recall but cannot recover every candidate missed
earlier. Multi-vector queries retain their mandatory original-vector refinement.
+
+#### When to Use Each Mode
+
+| Workload | Starting choice | What to verify |
+|---|---|---|
+| General workloads or an unknown bottleneck | `normal` | Establish recall and
latency before changing the policy. |
+| HNSW_FLAT with an effective prefilter, where latency matters | Compare
`fast` with `normal` | ACORN can reduce wasted distance calculations. Check
recall, result count, median latency, and p95; a speedup is not guaranteed. |
+| Multi-bit RabitQ where recall matters | `normal` | `fast` can discard useful
encoding information and substantially reduce recall. Use it only if
representative queries still meet the quality target. |
+| RabitQ without original-vector reranking, where approximate score error
matters | Compare `accurate` with `normal` | Measure score error and ranking
quality against original-vector distances, as well as extra latency. Better
estimation does not guarantee higher recall. |
+| RabitQ already using original-vector reranking | `normal` | The final score
benefit of `accurate` may be smaller; candidate selection can still differ, so
validate recall separately. |
+| IVF_FLAT or HNSW_FLAT without an effective prefilter | Keep `normal` | The
current implementation offers no corresponding quantization or ACORN benefit
from changing the mode. IVF_FLAT recall still depends on the probed partitions.
|
+
+For a compatible **HNSW index backed by RabitQ**, `fast` can both enable ACORN
traversal and simplify RabitQ distance estimation on the applicable raw-query
path. These effects are coupled by one option. Do not infer its recall from an
HNSW_FLAT test; validate the combined index independently. `normal` and
`accurate` have the same HNSW traversal policy, but their RabitQ scoring can
differ.
+
+Recommended tuning order:
+
+1. Pin the same dataset snapshot, query set, metric, filters, and index
coverage. Use `use_index=false` to obtain exact Top-K ground truth over the
same eligible rows; put eligibility conditions in the TVF `filter` with
`prefilter=true`.
+2. Establish a `normal` baseline. Measure Recall@K, result count, score error
where relevant, and median/p95 latency.
+3. If recall is insufficient, tune IVF `nprobes` and HNSW `ef`. If
original-vector scores or more reranked candidates are required, configure
`refine_factor`, accounting for the extra vector reads and computation.
+4. With the other settings fixed, compare `fast` for filtered HNSW or
`accurate` for RabitQ estimation precision. Select a mode only if it meets the
quality target and provides a worthwhile measured benefit. Measure cold and
warm cache behavior separately.
+
+#### Enable ACORN for Filtered HNSW
+
+The selected compatible index must contain an HNSW graph. Set
`approx_mode="fast"`, and place the predicate inside the TVF with
`prefilter="true"`. An outer Doris `WHERE` does not supply this prefilter, and
`use_index=true` alone does not force selection of an HNSW index.
+
+For the four-dimensional `embedding`, `user_id`, and string `category` columns
used in the syntax example, with an existing compatible L2 HNSW index and
`SELECT` privilege:
+
+```sql
+SELECT user_id, _distance
+FROM vector_search(
+ "table" = "lance_catalog.default.items",
+ "column" = "embedding",
+ "query_vector" = "[0.1,0.2,0.3,0.4]",
+ "metric" = "l2",
+ "top_k" = "10",
+ "filter" = "category = 'book'",
+ "prefilter" = "true",
+ "use_index" = "true",
+ "approx_mode" = "fast",
+ "ef" = "64"
+)
+ORDER BY _distance, user_id;
+```
+
+This returns up to ten eligible rows ranked by distance; the actual IDs and
recall depend on the data and ANN search. Running `EXPLAIN` for the query
includes `lanceApproxMode=FAST`, confirming the requested policy. It does not
prove that every subindex executed ACORN.
+
+The current native implementation dispatches separately inside each HNSW
subindex (an IVF partition), using the number of nodes passing the prefilter:
+
+| Runtime condition | Path |
+|---|---|
+| No effective prefilter, or all nodes pass | Ordinary HNSW traversal. |
+| Fewer than roughly 10% of nodes pass | Flat scoring over the remaining
nodes. |
+| A partial filter remains and the sparse-filter rule does not apply | `fast`
uses ACORN; `normal` and `accurate` use regular filtered graph traversal. |
+| ACORN returns too few required candidates and the query has no distance
bounds | Retry with regular filtered graph traversal. |
+
+The threshold applies within each subindex, not to the whole table. There is
no separate force-ACORN property in this interface. In Doris, distance-range
safety rules can also select Flat Search before native graph dispatch; see
[Radius and Range Search](#radius-and-range-search).
+
### Radius and Range Search
Set `distance_upper_bound` for a radius query, or combine both bounds for an
interval `[lower, upper)`. Omitted bounds are unbounded. Distances use the
selected metric: `l2` is **squared Euclidean distance**, so a geometric radius
of `r` corresponds to an upper bound of `r * r`. Negative bounds are accepted,
which is useful for metrics such as `dot`. Bounds use `FLOAT` precision even
when the stored vectors use Float64; non-finite bounds, overflow, equal bounds,
and reversed intervals ar [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
