danny0405 commented on issue #19215: URL: https://github.com/apache/hudi/issues/19215#issuecomment-4900414992
> After recovery, key=N is missing from the coordinator's buffer; on the next checkpoint completion the instant commits empty. The data/log files written for key=N are orphaned and never referenced → silent data loss / stuck inflight. it should be fine, the latter successful commits would clean the empty buffer. The real culprit might be from the logic in `AbstractStreamWriteFunction.sendBootstrapEvent`, previously we are assuming the coordinator does not restart during the task failure, so we skip to send the bootstrap event when attemptId > 0, but that is not true on global failover. The fix1 makes sense, it does not fix the data loss, it actaully avoid the repetitive commit of write stats. Fix2 is not necessary , for every recommit, the heartbeat will restart if the clean strategy is lazy. here is a patch, can you help for the data loss validation: [data_loss_on_global_failover.patch](https://github.com/user-attachments/files/29730115/data_loss_on_global_failover.patch) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
