wangyinsheng opened a new issue, #19339: URL: https://github.com/apache/hudi/issues/19339
### Describe the problem you faced
### Problem：
When enabling the Metadata Table (MDT) on existing Hudi tables that have
accumulated a large partition history, MDT bootstrap becomes a hard blocker for
the writer. In our production setup we routinely operate tables with the
following characteristics:
- Daily partitioning strategy (one partition per day)
- ~2 years of retained partitions (≈ 700+ partitions)
- Hundreds of thousands of files per partition
At this scale, the initial HoodieBackedTableMetadataWriter bootstrap
performs a full-table FILES (and optionally COLUMN_STATS / PARTITION_STATS /
BLOOM_FILTERS) scan across all historical partition before the writer is
allowed to resume normal ingestion. The consequences as follows:
1. The upstream writer (Flink / Spark streaming) has to be paused for a
very long window while MDT is being populated.
2. Business SLAs on ingestion latency are broken during the one-time MDT
rollout, which in practice discourages teams from turning MDT on at all.
### To Reproduce
1.
2.
3.
4.
### Expected behavior
### Proposal
We would like to open a discussion on a staged / phased MDT initialization
flow, roughly:
1. Phase 1 — Foreground bootstrap of hot partitions only.
During the initial write-side enablement, only bootstrap the partition(s)
that the current writer is actively producing into (e.g. today's partition, or
the last N partitions defined by a config knob). This should complete in
seconds/minutes and unblock the writer.
2. Phase 2 — Asynchronous / offline backfill of historical partitions.
Provide a Spark SQL CALL procedure that can be triggered independently —
e.g. `CALL run_metadata_backfill(table => '...', partitions => '...',
parallelism => ...)`. This job walks historical partitions and appends the
corresponding MDT records
incrementally, without blocking the online writer. It should be resumable,
idempotent, and safe to run concurrently with the writer (or at least with
well-defined isolation via existing lock providers / TxnManager).
3. Phase 3 — Reader activation gate.
MDT should stay invisible / disabled for readers until an explicit
"activation" step is performed (after Phase 2 completes). Concretely, readers
should either (a) fall back to filesystem listing for partitions that have not
yet been backfilled, or (b) treat MDT as unavailable until
a table-level flag / property (e.g. `hoodie.metadata.ready = true`) is
set. This avoids the pitfall of readers getting a partially-populated MDT view
during backfill and returning incorrect / incomplete results.
### Environment Description
* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):
### Additional context
_No response_
### Stacktrace
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
