vinothchandar opened a new pull request, #19179:
URL: https://github.com/apache/hudi/pull/19179
The Hudi 1.0 tech spec at /learn/tech-specs-1point0 hadn't been touched
since Dec 2024 and was scoped only to table version 8. Since then, Hudi 1.1.0
(Nov 2025) and Hudi 1.2.0 (May 2026) both shipped on table version 9, adding
new column types, a new base file format, and a new secondary-index encoding
without a further version bump. The document had also accumulated correctness
gaps (log block IDs listed as 1-based when they're serialized as zero-based
ordinals, missing metadata partitions, incomplete Avro schemas) that were
noticed while grounding the spec against the current code.
This change refreshes the spec end-to-end against the release-1.2.0 tag and
makes it the default tech spec on the site.
Scope
* Doc header now says table version 9, "Reflects release: Hudi 1.2.0", and
calls out that v9 readers cover the v8 subset but v8 readers can't in general
read v9-only features.
* Old table-version-5 spec (Aug 2022) is preserved at
/learn/tech-specs-0-point-x.
Correctness fixes
* Log block type IDs (Command, Delete, Corrupted, Avro, HFile, Parquet, CDC)
renumbered to zero-based (0-6) to match the enum ordinal that actually gets
serialized on the wire. Same for header metadata IDs.
* Added missing header metadata entry
BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS (ordinal 8).
* Storage-layout tree fixed: col_stats -> column_stats, added
partition_stats, bloom_filters, secondary_index_<name>, expr_index_<name>,
.temp/.
* Base file naming reconciled (requested_time consistent between template
and prose).
* Command block description says ROLLBACK_BLOCK (matches
HoodieCommandBlockTypeEnum) instead of the invented ROLLBACK_PREVIOUS_BLOCK.
* Typos: "Bboom filter", "fileIdLowBits, fileIdLowBits", missing "if" in
writer expectations, etc.
* Writer Expectations step "5. Writer checks" was dangling; now describes
the timeline refresh inside the lock.
Content completeness
* HoodieMetadataColumnStats table now enumerates all fields shipped in
HoodieMetadata.avsc: minValue, maxValue, valueCount, nullCount, totalSize,
totalUncompressedSize, isDeleted, isTightBound, valueType (HoodieValueTypeInfo).
* HoodieRecordIndexInfo table now includes the position field and a
paragraph on the partitioned-RLI file-id encoding.
* Bloom filter section calls out SIMPLE vs DYNAMIC.
* Secondary index section describes V1 (through v8) and V2 (default on v9)
encodings without dropping V1.
* Table Properties expanded from 11 to 26 rows to reflect the actual
hoodie.properties surface: create.schema, base.file.format, log.file.format,
keygenerator.class, cdc.enabled + supplemental logging mode,
partial.update.mode, metadata.partitions.inflight, timeline.path,
timeline.history.path, archivelog.folder, timeline.timezone, initial.version,
slash-separated date partitioning, and a note on atomic writes + checksum.
* Reader Expectations now describes rollback command blocks,
BLOCK_IDENTIFIER dedup, COMPACTED_BLOCK_TIMES skipping, and positional merges
anchored by BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS.
* Metadata section enumerates all seven metadata partitions and adds a
Partition Statistics subsection.
* Async indexer section replaces the undefined "t" with the actual
baseInstantTime from HoodieIndexPartitionInfo.
New content
* Data Files -> Types: short table covering the Hudi-specific logical types
added in table version 9 - VARIANT (record + logicalType variant), VECTOR
(fixed + logicalType vector), BLOB (record + logicalType blob).
* Base Files section explicitly lists Parquet, ORC, HFile, and Lance (Spark
COW/read only; column stats and partition stats not populated for Lance base
files).
* Change Log now has a "version 9 - current" section split into what landed
in Hudi 1.1.0 vs Hudi 1.2.0, plus updated version 8 highlights.
Site changes for the promotion
* website/learn/tech-specs.md is now the new (v9) spec; the old Aug 2022
spec moves to website/learn/tech-specs-0-point-x.md.
* Two self-references inside the promoted file were updated from
/learn/tech-specs (which used to point at the old spec) to
/learn/tech-specs-0-point-x. No other prose changes.
* sidebarsLearn.js: entries relabeled "Tech Specs (current)" and "Tech Specs
(pre-1.0)".
* docusaurus.config.js redirects: /tech-specs-1point0 and
/learn/tech-specs-1point0 both -> /learn/tech-specs so existing external links
(including versioned_docs and release notes on older sites) continue to
resolve. Footer link updated. Chat modal disclaimer URL updated.
* Live-tree markdown references (docs/hudi_stack.md,
blog/2025-04-02-secondary-index.md, releases/release-1.0*.md) rewritten from
tech-specs-1point0 -> tech-specs.
* versioned_docs/ left untouched by design - snapshots are frozen and the
redirect covers them.
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the
changes
(or) link to an issue by including `Closes #<issue-number>` for
context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in
behavior.
Followed by a detailed log of all the changes. Highlight if any code
was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any
performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain
the risk.
If medium or high, explain what verification was done to mitigate the
risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new
feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the
default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website.
Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website)
to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
