This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hudi-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 66ab307b docs: correct read config scoping and MOR streaming, document
reader v2 knobs (#709)
66ab307b is described below
commit 66ab307b5b64d4ec3e3a48191fda84c36ec856b2
Author: Y Ethan Guo <[email protected]>
AuthorDate: Thu Sep 3 15:09:28 2026 -0700
docs: correct read config scoping and MOR streaming, document reader v2
knobs (#709)
---
README.md | 43 +++++++++++++++++++++++++++++++++++++++----
docs/reader-spec.md | 11 ++++++-----
2 files changed, 45 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index aab198cd..e5a3ca6c 100644
--- a/README.md
+++ b/README.md
@@ -56,6 +56,13 @@ Hudi integration in the data ecosystems for a diverse range
of users and project
[crates]: https://crates.io/crates/hudi
[crates-badge]:
https://img.shields.io/crates/d/hudi?style=flat-square&color=163669
+The `hudi` crate carries two features: `datafusion` (off by default, see
+[Apache DataFusion](#apache-datafusion)) and `spill-rocksdb` (on by default),
the merge map's
+on-disk tier, which a merge-on-read merge spills to when a file group's log
records exceed
+`hoodie.memory.merge.max.size`. RocksDB is built from source with `bindgen`,
so it needs `libclang`
+and a C++ toolchain; `default-features = false` drops it, and a merge that
would have spilled then
+fails instead.
+
## Usage Examples
> [!NOTE]
@@ -331,7 +338,7 @@ All read APIs accept a `ReadOptions` (Rust) /
`HudiReadOptions` (Python) value.
- `batch_size` (`with_batch_size`) — rows per batch (streaming only; eager
reads return one batch per file slice).
- `as_of_timestamp` (`with_as_of_timestamp`) — snapshot/time-travel timestamp
(defaults to latest commit).
- `start_timestamp` / `end_timestamp` (`with_start_timestamp` /
`with_end_timestamp`) — incremental range (defaults to earliest…latest).
-- `hudi_options` — per-read Hudi configs (e.g.
`hoodie.read.use.read_optimized.mode`). Read configs are not stored in the
table; they flow exclusively through `ReadOptions`.
+- `hudi_options` — Hudi configs for this read (e.g.
`hoodie.read.use.read_optimized.mode`). A config that selects *which* read to
perform — `hoodie.read.query.type` and the as-of/start/end timestamps — is
per-read only and is dropped when set on the table. The rest describe *how* to
read: set them on the table and override them here. See [Read
configs](#read-configs).
| Stage | API
| Description
|
|-----------------|------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
@@ -340,6 +347,34 @@ All read APIs accept a `ReadOptions` (Rust) /
`HudiReadOptions` (Python) value.
| Query execution | `create_file_group_reader_with_options(read_options,
extra_storage_overrides)` | Create a file group reader with the table's
configs. Both args are optional. Timestamps are resolved automatically (e.g.
`AsOfTimestamp` → `EndTimestamp`), so callers can pass the same options used
for `get_file_slices`. |
| | `read(options)` / `read_stream(options)`
| Record-read APIs. Dispatch on `options.query_type`. `read_stream`
errors on `Incremental` for now. Per-slice streaming lives on
`FileGroupReader`. |
+### Read configs
+
+Read configs reach a read either through `ReadOptions` / `HudiReadOptions` or,
for those scoped
+`table or read`, through the table (`TableBuilder`, `hoodie.properties`,
`hudi-defaults.conf`), where
+a per-read value wins. A `per read` config set on the table is dropped: baked
in there, it would
+silently redirect every later read.
+
+| Config | Default
| Scope | Notes
|
+|-----------------------------------------------------------------|---------------|---------------|--------------------------------------------------------------------------------------------------------------------------|
+| `hoodie.read.query.type` | `snapshot`
| per read | `snapshot` or `incremental`.
|
+| `hoodie.read.as.of.timestamp` | latest
commit | per read | Snapshot time-travel point.
|
+| `hoodie.read.start.timestamp` / `hoodie.read.end.timestamp` | earliest /
latest | per read | Incremental window, half-open `(start, end]`.
|
+| `hoodie.read.file.group.reader.version` | `2`
| table or read | Which file group reader merges a slice. Version 2 is the
default; a read it cannot serve falls back to version 1. |
+| `hoodie.read.use.read_optimized.mode` | `false`
| table or read | Read base files only, skipping the log files, on MOR
tables. |
+| `hoodie.read.stream.batch_size` | `1024`
| table or read | Rows per batch for streaming reads.
|
+| `hoodie.read.file.slice.read.concurrency` | `4`
| table or read | File slices read concurrently.
|
+| `hoodie.read.scan.max.memory.size` | unset
| table or read | Total bytes a whole scan may use for concurrent slice
reads; when set, the concurrency is derived from it. Reaching the limit lowers
throughput, it never fails the read. |
+| `hoodie.read.input.partitions` | `0`
| table or read | How many partitions the DataFusion table provider buckets
the file slices into. `0` defers to DataFusion's `target_partitions`. |
+| `hoodie.merge.use.record.positions` | `false`
| table or read | Match a log record to the base row it updates by position
rather than by record key. Honored by reader version 2 only; a log block
written without positions is merged by key regardless. |
+
+A table whose `hoodie.record.merge.mode` is `CUSTOM` needs a merger for its
payload class. When
+neither reader has one, the read fails rather than returning wrong rows; set
+`hoodie.read.file.group.reader.version=1` to read it the way it was read
before, unless its base
+files are HFile, which version 1 cannot read.
+
+Base files are read from Parquet, Lance
(`hoodie.table.base.file.format=lance`), and HFile (metadata
+tables only).
+
### File Group API
Create a Hudi file group reader instance using its constructor or the Hudi
table API `create_file_group_reader_with_options()`.
@@ -348,13 +383,13 @@ Create a Hudi file group reader instance using its
constructor or the Hudi table
|-----------------|-----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Query execution | `read_file_slice()` | Read records
from a given file slice; based on the configs, read records from only base
file, or from base file and log files, and merge records based on the
configured strategy. |
| | `read_file_slice_from_paths()` | Read records
from an explicit base file path and a list of log file paths. Pass an empty log
path list to read just the base file.
|
-| | `read_file_slice_stream()` | Streaming
version of `read_file_slice()`. Yields true streaming batches when the slice is
base-file-only or read-optimized; for MOR slices with log files, falls back to
a single merged batch. |
+| | `read_file_slice_stream()` | Streaming
version of `read_file_slice()`. A base-file-only or read-optimized slice
streams straight from the base file. A MOR slice with log files streams merged
chunks under file group reader version 2 (the default), and collapses to a
single merged batch under version 1, whose merge has no incremental form. |
| | `read_file_slice_from_paths_stream()` | Streaming
version of `read_file_slice_from_paths()`.
|
### Apache DataFusion
-Enabling the `hudi` crate with `datafusion` feature will provide a
[DataFusion](https://datafusion.apache.org/)
+Enabling the `hudi` crate with `datafusion` feature will provide a
[DataFusion](https://datafusion.apache.org/)
extension to query Hudi tables.
<details>
@@ -408,7 +443,7 @@ table = HudiDataFusionDataSource(
"/tmp/trips_table", [("hoodie.read.input.partitions", "5")]
)
ctx = SessionContext()
-ctx.register_table_provider("trips", table)
+ctx.register_table("trips", table)
ctx.sql("SELECT max(fare), city from trips group by city order by 1
desc").show()
```
diff --git a/docs/reader-spec.md b/docs/reader-spec.md
index 07ace4ac..54405f12 100644
--- a/docs/reader-spec.md
+++ b/docs/reader-spec.md
@@ -79,7 +79,7 @@ Which knobs each API consumes:
Notes:
- `read_stream` errors with `Unsupported` for `query_type = Incremental` —
incremental streaming is not yet implemented.
-- The `hudi_options` bag is a per-read override layer — set arbitrary
`hoodie.read.*` configs (e.g. `hoodie.read.use.read_optimized.mode = true`) for
this single read. Read configs (`hoodie.read.*`) are not stored in the `Table`
instance; they flow exclusively through `ReadOptions`.
+- The `hudi_options` bag is a per-read override layer — set arbitrary
`hoodie.read.*` configs (e.g. `hoodie.read.use.read_optimized.mode = true`) for
this single read. Each read config declares a scope. A `ReadOnly` one —
`hoodie.read.query.type` and the as-of/start/end timestamps — selects *which*
read to perform, so it flows exclusively through `ReadOptions` and is dropped
when set on the table. A `TableOrRead` one describes *how* to read, so it may
be set on the table and overridden h [...]
- Per-slice reads are exposed only by `FileGroupReader`. The `Table` type owns
logical reads (snapshot, incremental); per-slice reads are physical and belong
at the file-group layer. To read one slice with table-level configs, build a
`FileGroupReader` via `Table::create_file_group_reader_with_options` and call
its per-slice methods. The method resolves timestamps automatically (e.g.
`AsOfTimestamp` → `EndTimestamp`), so callers can pass the same `ReadOptions`
used for `get_file_slices`.
- For parallel reads, call `get_file_slices(...)` and bucket the result with
`hudi::util::collection::split_into_chunks` or your engine's preferred
partitioning policy.
@@ -102,6 +102,7 @@ The `field` may be any column. Filters drive three things:
- **Partition pruning** when the field is a partition column. Always applied.
- **File-level stats pruning** when the field is a data column with min/max
stats in the metadata table. Snapshot/time-travel only — incremental file
planning does not stats-prune.
+- **Row-group pruning** inside a base file, from the parquet footer's own
statistics. Conservative: a row group that cannot match is never fetched, and
one that might match is kept.
- **Row-level mask** applied to every returned batch. This is the
authoritative filter; pruning is best-effort.
Values are strings; they are cast to the target column's Arrow type at filter
time. Unparseable values (e.g. `"abc"` against `Int64`) error.
@@ -176,7 +177,7 @@ let table =
HudiTableBuilder::from_base_uri("/tmp/trips_table")
.await?;
```
-Available pairs: `with_hudi_option` / `with_hudi_options`,
`with_storage_option` / `with_storage_options`, `with_option` / `with_options`
(the generic forms route by key prefix). Read configs (`hoodie.read.*`) passed
at table construction are silently dropped — they belong in `ReadOptions`
per-call.
+Available pairs: `with_hudi_option` / `with_hudi_options`,
`with_storage_option` / `with_storage_options`, `with_option` / `with_options`
(the generic forms route by key prefix). A read config passed at table
construction is kept when its scope is `TableOrRead`
(`hoodie.read.file.group.reader.version`,
`hoodie.read.use.read_optimized.mode`, `hoodie.read.stream.batch_size`,
`hoodie.read.file.slice.read.concurrency`, `hoodie.read.scan.max.memory.size`,
`hoodie.read.input.partitions`, `hood [...]
### Filter, Timeline, FileSlice
@@ -214,7 +215,7 @@ table = (
)
```
-`with_hudi_option` and `with_option` accept a string key or a `HudiReadConfig`
/ `HudiTableConfig` enum member. The bulk variants (`with_hudi_options`,
`with_options`) currently accept dicts of string keys. Read configs
(`hoodie.read.*`) are silently dropped at table construction — pass them via
`HudiReadOptions` per-call instead.
+`with_hudi_option` and `with_option` accept a string key or a `HudiReadConfig`
/ `HudiTableConfig` enum member. The bulk variants (`with_hudi_options`,
`with_options`) currently accept dicts of string keys. Read configs scoped
`TableOrRead` are kept at table construction and overridden per read; the
`ReadOnly` ones (`hoodie.read.query.type` and the as-of/start/end timestamps)
are dropped there — pass them via `HudiReadOptions` per-call instead. See
[§2](#2-readoptions).
### `HudiTable`
@@ -304,7 +305,7 @@ The range is half-open: (`start_timestamp`,
`end_timestamp`]. A record updated m
### MOR streaming fallback
-Streaming yields true streaming batches when the slice is base-file-only or
`hoodie.read.use.read_optimized.mode = true`. For MOR slices with log files,
the implementation collects-and-merges and yields the result as a single batch
on the stream.
+Streaming yields true streaming batches when the slice is base-file-only or
`hoodie.read.use.read_optimized.mode = true`. A MOR slice with log files is
merged, and what the stream yields depends on which file group reader serves
it: version 2, the default, merges a bounded chunk at a time and yields each
chunk, while version 1 has no incremental form for its whole-batch sort and
dedup, so it collects-and-merges and yields the result as a single batch.
### `batch_size` and `projection`
@@ -342,6 +343,6 @@ Reader APIs documented here are the supported public
surface as of this release.
For I/O cost estimation (on-disk base + log file sizes), use
`FileSlice::total_size_bytes()` instead.
-`Table` / `HudiTable` only stores table configs (`HudiTableConfig`). Read
configs (`HudiReadConfig`, keyed under `hoodie.read.*`) are filtered out during
construction and flow exclusively through `ReadOptions` / `HudiReadOptions`
per-call. `hudi_options()` on the table reflects the stored table configs, not
any read configs the caller may have passed at construction.
+`Table` / `HudiTable` stores table configs (`HudiTableConfig`) and those read
configs whose scope is `TableOrRead`; a per-read value overrides them.
`ReadOnly` read configs — `hoodie.read.query.type` and the as-of/start/end
timestamps — are filtered out during construction and flow exclusively through
`ReadOptions` / `HudiReadOptions` per-call, so `hudi_options()` on the table
never reflects them.
Out of scope for this version: writer APIs, internal architecture (timeline
parsing, log-record merging, metadata table layout), the full configuration key
glossary (`HudiReadConfig` / `HudiTableConfig` members), and the DataFusion and
C++ bindings — separate spec follow-ups.
