nsivabalan opened a new pull request, #19205:
URL: https://github.com/apache/hudi/pull/19205
### Describe the issue this Pull Request addresses
Today the only way to opt out of populating Hudi's five meta columns is the
all-or-nothing `hoodie.populate.meta.fields=false`. That saves storage but
disables incremental queries (which require `_hoodie_commit_time`) and
file-level pruning / investigation lookups (which need `_hoodie_file_name`).
A community user surfaced this trade-off (#18383, also discussed at #17959).
This PR proposes a scoped, list-based property that lets a table opt into
`_hoodie_commit_time` and/or `_hoodie_file_name` without paying for the
remaining three meta columns.
This PR supersedes #19047 (the interim
`hoodie.meta.fields.commit.time.enabled` boolean approach). Closes #18383.
### Summary and Changelog
Adds a new table property `hoodie.meta.fields.mode` — a comma-separated list
of meta columns to populate when `hoodie.populate.meta.fields=false`. Allowed
tokens are `_hoodie_commit_time` and `_hoodie_file_name`; any other token is
rejected up-front. The mode is immutable at runtime (settable only at table
creation, via hudi-cli, or during upgrade).
Resulting modes:
| \`populate.meta.fields\` | \`meta.fields.mode\` | Effective mode |
|---|---|---|
| \`true\` (default) | ignored | **ALL** — today's default |
| \`false\` | empty | **NONE** — today's minimal-meta-fields |
| \`false\` | \`_hoodie_commit_time\` | **COMMIT_TIME_ONLY** |
| \`false\` | \`_hoodie_file_name\` | **FILE_NAME_ONLY** |
| \`false\` | \`_hoodie_commit_time,_hoodie_file_name\` |
**COMMIT_TIME_AND_FILE_NAME** |
| \`true\` | non-empty | rejected at writer init (ambiguous) |
#### Why a list instead of a boolean or enum
- Extensible without another config flag — adding a third opt-in field later
means changing the allowed set, not adding a new property.
- Empty default preserves bit-identical backward compat — every existing
table on disk resolves to ALL or NONE without any new property being read.
- Pre-1.3.0 readers see \`populate.meta.fields=false\` and treat the table
as NONE — they cannot do incremental queries on the table, but they don't
produce silent wrong results either.
- Structurally encodes "additive on NONE" — most code paths that branch on
\`populate.meta.fields\` keep working unchanged; only paths that specifically
need commit_time / file_name consult the new accessors.
#### Plug points
**Config + accessors (\`hudi-common\` / \`hudi-client-common\`):**
- New \`HoodieTableConfig.META_FIELDS_MODE\` property + static
\`parseMetaFieldsMode(String)\` helper.
- New predicates: \`isCommitTimePopulated()\`, \`isFileNamePopulated()\`,
\`isRecordKeyPopulated()\`.
- \`HoodieWriteConfig\` pass-through accessors +
\`Builder.withMetaFieldsMode(String)\`.
- \`HoodieWriteConfig.validate()\` rejects:
- unknown token in the mode list (parser throws),
- \`populate=true\` + non-empty mode (ambiguous),
- \`MERGE_ON_READ\` + non-empty mode (log-write path not yet wired;
follow-up),
- non-Spark engine + non-empty mode (Flink RowData / Java-client not yet
wired; follow-up).
- \`HoodieTableMetaClient.TableBuilder.setMetaFieldsMode(String)\` persists
the mode on \`hoodie.properties\` at table init.
- \`HoodieSparkSqlWriter\` wires both fresh-table and bootstrap creation
paths.
**Writer engines:**
- \`HoodieAvroParquetWriter\`, \`HoodieSparkParquetWriter\`,
\`HoodieRowCreateHandle\` each take a \`Set<String> metaFieldsMode\` and
populate columns selectively. Bloom filter / record-key index registration is
intentionally skipped when the record-key column is not populated.
**Read path (incremental query rejection):**
- \`IncrementalRelationV1/V2\` (CoW) now check \`isCommitTimePopulated()\`
rather than \`populateMetaFields()\` — COMMIT_TIME_ONLY tables are accepted,
NONE tables remain rejected with a clearer message.
- \`MergeOnReadIncrementalRelationV1/V2\` (MoR) keep the strict
\`populateMetaFields()\` guard until the MoR log-write path is updated.
Selective modes are CoW-only in this release; MoR support is a follow-up.
**Runtime immutability guard (\`HoodieWriterUtils.validateTableConfig\`):**
- Explicit null-on-disk → non-empty-in-write-options transition on the mode
property is now blocked with a message directing the user to CLI / upgrade. The
existing loop only flagged the mismatch when the on-disk value was non-null,
which let a silent-drop path slip through for older tables.
#### Scope
- ✅ Spark Avro / Spark Row writer paths.
- ✅ Spark Parquet bulk-insert.
- ✅ CoW incremental query rejection logic across V1 / V2.
- ❌ MoR — \`HoodieAppendHandle.populateMetadataFields\` is not yet updated
for the new mode, so MoR + non-empty mode is rejected at writer init. Tracked
as a follow-up PR.
- ❌ Flink RowData / Java-client writers — out of scope for this patch; the
writer init rejects non-empty mode with a clear message. Tracked as a follow-up
PR.
- ❌ ORC / HFile writers — ORC continues to populate all meta fields
unconditionally (legacy behavior); HFile is used only by MDT which is always
ALL.
### Impact
- **Storage layout**: no change for tables that don't opt in. New optional
mode for tables that do. Default behavior unchanged.
- **API**: no public API breakage. New table property, new accessors, new
builder method — all additive.
- **Configuration**:
- \`hoodie.meta.fields.mode\` (default \`""\`). Only meaningful when
\`hoodie.populate.meta.fields=false\`. Persisted on \`hoodie.properties\` at
table init. Immutable at runtime.
- **Performance**: writer hot path adds one Set membership check per row per
opt-in field. The set is parsed once in the writer constructor.
- **Forward-compat**: pre-1.3.0 readers ignore the new property and treat
the table as NONE — no silent wrong results.
### Risk Level
low
Additive change with a narrow scope. The default path is untouched.
Fail-fast validation at writer init rejects every ambiguous combination loudly.
Runtime property changes are blocked.
### Documentation Update
- New config \`hoodie.meta.fields.mode\` documented via \`@ConfigProperty\`
annotation on \`HoodieTableConfig\`.
- If the website page on meta fields exists, a separate docs PR will add the
new mode table.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
