corgy-w opened a new issue, #12439: URL: https://github.com/apache/seatunnel/issues/12439
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description The Hudi sink commits every flushed batch with the Hudi client auto-commit, so the records become visible before the checkpoint that contains them completes. When a job fails and the source replays the records of a checkpoint that never completed, those records are committed to Hudi a second time. The sink therefore only provides at-least-once delivery for Zeta, see `docs/en/connectors/sink/Hudi.md` and the earlier timer flush work in #11747. It would be useful for the sink to support the two-phase commit protocol of the engine, so that the records of a checkpoint are committed to Hudi only after that checkpoint completes, which gives exactly-once delivery. Hudi itself supports this pattern: the data is written into an instant while the job is running, and the instant is committed afterwards, so readers only see completed instants. ### Usage Scenario * A streaming job replicates data from a CDC/JDBC source into Hudi and must not duplicate records when the job fails and restarts from a checkpoint. * A batch job writes into Hudi and needs the whole checkpoint to be published atomically instead of batch by batch. ### Related issues * #11747 timer flush of the Hudi sink -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
