danny0405 opened a new pull request, #19067: URL: https://github.com/apache/hudi/pull/19067
### Describe the issue this Pull Request addresses This closes #19058 . RFC-103 introduces an LSM-tree table storage layout that needs MOR append writes to produce native-format log files instead of inline Hudi log blocks. The existing append path was centered on `HoodieAppendHandle` buffering records into inline log blocks, which made it hard to support engine-native record types, native file writers, native delete logs, footer metadata, and per-rolled-file column stats. This PR adds the opt-in table config `hoodie.table.storage.layout=lsm_tree` while keeping the default layout unchanged. When the layout is `lsm_tree`, MOR append/update/log-compaction paths route to native log append handles that write native data and delete log files, including native log footer metadata with format version `2`. ### Summary and Changelog This PR refactors append handling into a shared append base plus inline/native implementations, then wires Spark, Java, and Flink MOR write paths to choose inline or native log handles based on the table storage layout. #### Working tree: RFC-103 native log append support - Added `hoodie.table.storage.layout` in `HoodieTableConfig` with values `default` and `lsm_tree`; `default` preserves the current layout. - Split `HoodieAppendHandle` into an abstract common append lifecycle and `HoodieInlineLogAppendHandle`, preserving the existing inline log behavior behind the inline implementation. - Added `HoodieNativeLogAppendHandle` and `HoodieNativeLogFormatWriter` to stream records through engine-native file writers instead of collecting records into inline data/delete blocks. - Added native data and delete log handling, including `.log.parquet` and `.deletes.parquet` files, native delete-log row construction, native footer metadata under `hudi.log.format.metadata`, and native log format `VERSION=2`. - Routed `AppendHandleFactory`, Spark/Java MOR delta executors, Flink write handle factories, and log compaction to select native append handles when `isLSMTreeStorageLayout()` is enabled. - Added native variants for file-group-reader compaction and Flink/RowData append paths: `FileGroupReaderBasedNativeLogAppendHandle`, `FlinkNativeLogAppendHandle`, and `RowDataNativeLogWriteHandle`. - Renamed existing append classes to make inline/native behavior explicit: `FileGroupReaderBasedInlineLogAppendHandle`, `FlinkInlineLogAppendHandle`, and `RowDataInlineLogWriteHandle`. - Added file-writer footer metadata plumbing so native log column stats can be collected from native file footers rather than iterating buffered records. - Updated RFC-103 documentation and existing tests touched by the append-handle split. #### Working tree: log extension constants and comparator coverage - Added `LogExtensions` with shared `DATA_LOG_EXTENSION`, `DELETE_LOG_EXTENSION`, `CDC_LOG_EXTENSION`, and `ARCHIVE_LOG_EXTENSION` constants. - Updated `HoodieLogFile` extension precedence and `HoodieNativeLogFormatWriter` to use `LogExtensions` instead of duplicated string literals. - Added `TestHoodieLogFile.logFileComparatorOrdersNativeLogExtensionsByPrecedenceForSameVersion` to pin native log ordering as `log < deletes < cdc < archive`. ### Impact This is an opt-in storage-layout change. Existing tables continue to use the default inline log layout unless `hoodie.table.storage.layout=lsm_tree` is configured. The PR affects core MOR append/update/log-compaction paths across Spark, Java, and Flink, plus log file naming/comparison and metadata-column-stats collection for native logs. Native logs introduce a new physical log representation and footer metadata contract, but the default layout remains backward compatible. ### Risk Level medium The risk is medium because the change touches core MOR write paths, append-handle lifecycle accounting, log file naming/comparison, Flink RowData write handles, log compaction, native file-writer metadata, and column-stats collection. The main compatibility mitigation is that the new behavior is gated behind `hoodie.table.storage.layout=lsm_tree`, with `default` retaining the current inline log layout. Validation evidence: - `git diff --check` passed for the latest touched files. - Attempted `mvn -pl hudi-client/hudi-client-common -DskipTests compile`, but it is currently blocked by an existing compile error in `HoodieNativeLogFormatWriter.java` resolving `org.apache.hudi.common.schema.HoodieSchemas`. - Attempted `mvn -pl hudi-common -DskipTests compile` and `mvn -pl hudi-common -DskipITs -DfailIfNoTests=false -Dtest=TestHoodieLogFile test`, but both are currently blocked by an existing compile error in `HoodieSchema.java` resolving `StringUtils.splitTopLevelCommas(...)`. ### Documentation Update RFC-103 is updated in this PR. Additional user-facing docs may be needed for the new `hoodie.table.storage.layout=lsm_tree` config before enabling or advertising the layout outside the RFC context. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
