chrevanthreddy commented on code in PR #19309: URL: https://github.com/apache/hudi/pull/19309#discussion_r3677433282
########## rfc/rfc-109/rfc-109.md: ########## @@ -0,0 +1,594 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-109: Native Vector Search Support in Apache Hudi + +## Proposers + +@chrevanthreddy + +## Approvers + +- TBD + +## Status + +Umbrella issue: [apache/hudi#19094](https://github.com/apache/hudi/issues/19094) + +Related: [apache/hudi#18676](https://github.com/apache/hudi/issues/18676) + +State: UNDER REVIEW + +--- + +## Table of Contents + +- [Abstract](#abstract) +- [1. Goals and Non-Goals](#1-goals-and-non-goals) +- [2. Architecture](#2-architecture) +- [3. IVF + RaBitQ Index Algorithm](#3-ivf--rabitq-index-algorithm) +- [4. Metadata Table Storage Model: the Posting Block](#4-metadata-table-storage-model-the-posting-block) +- [5. Bootstrap and Write Path](#5-bootstrap-and-write-path) +- [6. Read Path](#6-read-path) +- [7. Maintenance, Rebalancing, and Cleaner](#7-maintenance-rebalancing-and-cleaner) +- [8. Spark API Surface](#8-spark-api-surface) +- [9. Correctness and Consistency](#9-correctness-and-consistency) +- [10. Test Plan](#10-test-plan) +- [11. Rollout and MVP Scope](#11-rollout-and-mvp-scope) +- [12. References](#12-references) + +--- + +## Abstract + +This RFC proposes native approximate nearest-neighbor (ANN) vector search in Apache Hudi. +Tables increasingly carry embedding columns (`ARRAY<FLOAT>` produced by ML models) next to +their business data, and users want to ask *"find the K rows most similar to this query +vector"* — for semantic search, recommendations, RAG, and deduplication — without copying +data into a separate vector database. + +Today the only option on a Hudi table is a brute-force scan: read every vector, compute +every distance. That is correct but scales linearly with table size (tens of seconds at a +billion rows). This RFC adds an index so that vector queries read only a small, targeted +fraction of the index and the table, return results with high recall, and stay +transactionally consistent with the table under upserts and deletes — all with **no new +storage system**. The index lives in the Hudi Metadata Table (MDT), like Hudi's existing +record-level and secondary indexes, and is maintained by the same table services. + +The design combines three well-understood pieces — IVF clustering, RaBitQ quantization, and +exact re-ranking — with one storage innovation that makes them practical on an immutable, +columnar, object-store-resident lakehouse: + +> **The posting block.** Instead of one MDT record per indexed vector, the index packs +> ~1–4K vectors into a single MDT record laid out column-wise (structure-of-arrays), keyed +> so that one IVF cluster forms one contiguous, prefix-scannable key range. This reduces MDT +> record count by roughly three orders of magnitude, turns "scan a cluster" into a single +> contiguous range read, and lets a query touch only the columns a given scan pass needs. + +The base table remains the source of truth for exact vector values. The MDT stores only +routing, pruning, and approximate-scoring metadata; final ranking always reads exact vectors +from the base table. + +Prototype measurements on a **1-billion-row, 128-dimensional table** show exact-reranked +recall@10 = 0.985 at nprobe=128 with query latency in low single-digit seconds on a modest +Spark cluster, versus ~15s for brute force. + +--- + +## 1. Goals and Non-Goals + +### 1.1 Goals + +1. Keep authoritative vector values in the base table (`ARRAY<FLOAT>` / `VECTOR(D)` column). +2. Store the vector index in the MDT, maintained by Hudi metadata-table commits, compaction, + and cleaning — no hidden or generated columns in base-table files. +3. Make candidate discovery cheap and targeted: probe a few clusters, scan contiguous key + ranges, score on compressed codes with a provable pruning bound. +4. Make results trustworthy: approximate math selects candidates; **exact** distance on + base-table vectors ranks them. +5. Stay transactionally consistent: snapshot-pinned reads, correct behavior under inserts, + updates, deletes, and clustering. +6. Be engine-neutral in design; Spark is the first implementation. +7. Maintain the index incrementally (no global rebuild for normal churn) and support + versioned, zero-downtime rebuilds. + +### 1.2 Non-Goals (initial landing) + +- ANN families beyond IVF + RaBitQ (e.g. HNSW, DiskANN). +- Filtered search (arbitrary predicate + kNN) as a first-class planned operation. +- Time-travel-consistent index reads for historical snapshots. +- Engines beyond Spark, and GPU-accelerated encoding. +- Workload-specific auto-tuning of `nprobe` / refine factor. + +--- + +## 2. Architecture + + + +The design splits responsibilities the way Hudi already does between the data table and the +metadata table: + +```text +DATA TABLE (parquet/orc) METADATA TABLE (vector_index partition) + authoritative vectors + payload ←── the index: centroids, quantizer, posting blocks, + read only for final re-ranking cluster manifests, generation manifest + read for candidate generation +``` + +Each vector index is one MDT partition. Creating an index: Review Comment: Addressed in §2, §4.2, and §9.1 in commit `b902152fa6`. The vector-index partition follows the existing MDT layer-2 compatibility model: readers and writers that do not understand it ignore it rather than failing ordinary table access. RFC-109 does not add a table-level required-features property or change unrelated write paths. Vector-aware readers use the manifest's persisted-format version and reject unsupported or mixed generation formats. Freshness is enforced by the vector planner rather than by trusting every writer. For each data-write instant after the bootstrap baseline, a feature-aware hook writes `F|generation|dataInstant` in the same MDT commit as its vector deltas; when there are no vector deltas, the marker itself is the non-empty update. At planning, Hudi derives the contiguous marker frontier from the active data-write timeline and compares it with the pinned instant. A legacy writer therefore leaves a detectable, repairable gap; `FAIL`, `WARN`, or exact `FALLBACK` applies until catch-up fills it. The revision also makes the column contract explicit. RFC-109 consumes RFC-99's top-level `VECTOR(D[, elementType])`; the table schema is authoritative and the generation repeats dimension/type only for integrity validation. The current RFC-99 backing is Avro `FIXED` / Parquet `FIXED_LEN_BYTE_ARRAY(D × elementWidth)`, while Spark exposes an annotated `ArrayType` and converts at the boundary. A plain `ARRAY<FLOAT>` is not reinterpreted in place and requires explicit migration/backfill before index creation. ########## rfc/rfc-109/rfc-109.md: ########## @@ -0,0 +1,594 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-109: Native Vector Search Support in Apache Hudi + +## Proposers + +@chrevanthreddy + +## Approvers + +- TBD + +## Status + +Umbrella issue: [apache/hudi#19094](https://github.com/apache/hudi/issues/19094) + +Related: [apache/hudi#18676](https://github.com/apache/hudi/issues/18676) + +State: UNDER REVIEW + +--- + +## Table of Contents + +- [Abstract](#abstract) +- [1. Goals and Non-Goals](#1-goals-and-non-goals) +- [2. Architecture](#2-architecture) +- [3. IVF + RaBitQ Index Algorithm](#3-ivf--rabitq-index-algorithm) +- [4. Metadata Table Storage Model: the Posting Block](#4-metadata-table-storage-model-the-posting-block) +- [5. Bootstrap and Write Path](#5-bootstrap-and-write-path) +- [6. Read Path](#6-read-path) +- [7. Maintenance, Rebalancing, and Cleaner](#7-maintenance-rebalancing-and-cleaner) +- [8. Spark API Surface](#8-spark-api-surface) +- [9. Correctness and Consistency](#9-correctness-and-consistency) +- [10. Test Plan](#10-test-plan) +- [11. Rollout and MVP Scope](#11-rollout-and-mvp-scope) +- [12. References](#12-references) + +--- + +## Abstract + +This RFC proposes native approximate nearest-neighbor (ANN) vector search in Apache Hudi. +Tables increasingly carry embedding columns (`ARRAY<FLOAT>` produced by ML models) next to +their business data, and users want to ask *"find the K rows most similar to this query +vector"* — for semantic search, recommendations, RAG, and deduplication — without copying +data into a separate vector database. + +Today the only option on a Hudi table is a brute-force scan: read every vector, compute +every distance. That is correct but scales linearly with table size (tens of seconds at a +billion rows). This RFC adds an index so that vector queries read only a small, targeted +fraction of the index and the table, return results with high recall, and stay +transactionally consistent with the table under upserts and deletes — all with **no new +storage system**. The index lives in the Hudi Metadata Table (MDT), like Hudi's existing +record-level and secondary indexes, and is maintained by the same table services. + +The design combines three well-understood pieces — IVF clustering, RaBitQ quantization, and +exact re-ranking — with one storage innovation that makes them practical on an immutable, +columnar, object-store-resident lakehouse: + +> **The posting block.** Instead of one MDT record per indexed vector, the index packs +> ~1–4K vectors into a single MDT record laid out column-wise (structure-of-arrays), keyed +> so that one IVF cluster forms one contiguous, prefix-scannable key range. This reduces MDT +> record count by roughly three orders of magnitude, turns "scan a cluster" into a single +> contiguous range read, and lets a query touch only the columns a given scan pass needs. + +The base table remains the source of truth for exact vector values. The MDT stores only +routing, pruning, and approximate-scoring metadata; final ranking always reads exact vectors +from the base table. + +Prototype measurements on a **1-billion-row, 128-dimensional table** show exact-reranked +recall@10 = 0.985 at nprobe=128 with query latency in low single-digit seconds on a modest +Spark cluster, versus ~15s for brute force. + +--- + +## 1. Goals and Non-Goals + +### 1.1 Goals + +1. Keep authoritative vector values in the base table (`ARRAY<FLOAT>` / `VECTOR(D)` column). +2. Store the vector index in the MDT, maintained by Hudi metadata-table commits, compaction, + and cleaning — no hidden or generated columns in base-table files. +3. Make candidate discovery cheap and targeted: probe a few clusters, scan contiguous key + ranges, score on compressed codes with a provable pruning bound. +4. Make results trustworthy: approximate math selects candidates; **exact** distance on + base-table vectors ranks them. +5. Stay transactionally consistent: snapshot-pinned reads, correct behavior under inserts, + updates, deletes, and clustering. +6. Be engine-neutral in design; Spark is the first implementation. +7. Maintain the index incrementally (no global rebuild for normal churn) and support + versioned, zero-downtime rebuilds. + +### 1.2 Non-Goals (initial landing) + +- ANN families beyond IVF + RaBitQ (e.g. HNSW, DiskANN). +- Filtered search (arbitrary predicate + kNN) as a first-class planned operation. +- Time-travel-consistent index reads for historical snapshots. +- Engines beyond Spark, and GPU-accelerated encoding. +- Workload-specific auto-tuning of `nprobe` / refine factor. + +--- + +## 2. Architecture + + + +The design splits responsibilities the way Hudi already does between the data table and the +metadata table: + +```text +DATA TABLE (parquet/orc) METADATA TABLE (vector_index partition) + authoritative vectors + payload ←── the index: centroids, quantizer, posting blocks, + read only for final re-ranking cluster manifests, generation manifest + read for candidate generation +``` + +Each vector index is one MDT partition. Creating an index: + +```sql +CREATE INDEX embedding_idx +ON products +USING VECTOR (embedding) +OPTIONS ( + 'vector.dimension' = '768', + 'vector.metric' = 'cosine', + 'vector.quantizer' = 'IVF_RABITQ', + 'vector.num_clusters'= '4096' +); +``` + +creates: + +```text +.hoodie/metadata/vector_index_embedding_idx/ +``` + +and does not change the base-table schema: + +```text +products/category=electronics/<file-group>.parquet +├── _hoodie_record_key = p001 +├── _hoodie_partition_path = category=electronics +├── user columns = ... +└── embedding ARRAY<FLOAT> = [0.12, -0.08, ...] +``` + +The query path uses the MDT first to discover candidates, then reads base-table vectors for +exact re-ranking: + +```text +query vector + → compare to centroids, pick nprobe clusters (in-memory, ms) + → MDT prefix-scan those clusters' posting blocks (targeted range reads) + → two-pass RaBitQ scoring, keep refineFactor·K best (bit math + error bounds) + → validate candidate freshness via Record Level Index (batched point lookups) + → fetch ONLY those rows from the base table by position (page-level reads) + → exact distance on real vectors → final top-K +``` + +--- + +## 3. IVF + RaBitQ Index Algorithm + +Every practical ANN index answers two questions: **where to look** (avoid scanning +everything) and **how to compare cheaply** (avoid full-precision math on what is scanned). + +### 3.1 Where to look: IVF routing + +Inverted File (IVF) indexing clusters vectors with KMeans into `numClusters` groups (e.g. +~4K–64K). Each vector belongs to its nearest centroid. A query compares against the centroids +only (thousands, not billions), selects the `nprobe` nearest clusters, and scans only those +clusters' entries. `nprobe` is the recall dial. + +IVF is the right fit for a lakehouse-resident index because a cluster's entries can be stored Review Comment: Addressed in §1.3 in commit `b902152fa6`. The revision now evaluates the higher-level choices explicitly: * one MDT record per vector is simple but creates prohibitive record-count, write, compaction, and scan amplification; * dedicated index files permit specialized layouts but require a second commit/cleaning/snapshot protocol; * external sidecars or vector databases may provide richer serving features, but lose atomic Hudi snapshot semantics and add another storage system; * native ANN libraries can improve local kernels, but do not define the durable object-store layout, multi-writer maintenance, or snapshot-visibility protocol. The proposed posting-block layout keeps lifecycle and visibility in MDT while amortizing per-record overhead. Native libraries remain usable behind the encoder/scorer interfaces without changing the persisted contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
