yihua opened a new pull request, #20261: URL: https://github.com/apache/hudi/pull/20261
### Describe the issue this Pull Request addresses closes #20260 part of #20257 With `hoodie.allow.operation.metadata.field=true`, MOR upserts and deletes from writers that do not set a record operation (Spark datasource, Spark SQL, Java client) NPE on every record in `HoodieAppendHandle.populateMetadataFields` on table versions 6, 8 and 10. The failure is swallowed under the default `hoodie.write.ignore.failed=true`, and `close()` drops the all-failed write status, so the deltacommit succeeds without the records. The NPE dates from #3285 (0.9.0), and the dropped status from #2300 (0.7.0). ### Summary and Changelog MOR writes with the operation field enabled now land in the log files, and record failures in an append handle reach the caller. - `populateMetadataFields` derives `U`/`I` from the record location when the record has no operation. Log compaction keeps the stored value. - `writeRecord` sends a delete to the delete block unless it carries both an operation and a row. - `close()` returns the write status of a handle that failed without appending a block, and sets `totalWriteErrors` on the stats so table services fail on record errors. - Write stats without a path are skipped in `CommitUtils.buildMetadata` and `HoodieTablePreCommitFileSystemView`, and `CoalescingPartitioner` maps a null key to partition 0. - Tests: `TestMORDataSource` (table versions 6, 8 and 10, upsert, delete, inline compaction; red on master), plus unit tests in `TestHoodieNativeLogAppendHandle`, `TestCommitUtils`, `TestHoodieTablePreCommitFileSystemView` and `TestCoalescingPartitioner`. ### Impact Fixes silent data loss for MOR writes with the operation field enabled from non-Flink writers. Log compaction with record failures now fails instead of committing without those records. ### Risk Level medium: writes that used to lose records silently now report the failures, so Spark datasource, SQL and Streamer writes with failed records fail or roll back unless configured to commit despite errors. Spark and Java log compaction now fails on any record failure instead of committing without those records (Flink log compaction still commits, as its error handling is a no-op). Write stats without a path are dropped when commit metadata is assembled and skipped in pre-commit validation. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
