K1taIkuyo0421 commented on issue #20038: URL: https://github.com/apache/hudi/issues/20038#issuecomment-5796507187
Hi @danny0405 , I'd also like to discuss the incomplete `eventBuffer` after rescaling reported in [#19922](https://github.com/apache/hudi/issues/19922). You mentioned in [#19952](https://github.com/apache/hudi/pull/19952) that this would be addressed later. I have implemented a draft solution: - Every Writer subtask sends a bootstrap event during recovery. It sends the actual metadata event when restored metadata exists, or a placeholder event when there is no restored metadata. - Each event carries the `restoredCheckpointId`. The Coordinator maintains a `Map<Long, boolean[]>` to track the arrival of events from each subtask according to the `restoredCheckpointId`. - The actual metadata remains in the original `eventBuffers`. The auxiliary buffer is only used to determine whether all events have arrived. Once all events are collected, the Coordinator performs the recommit. If all events are placeholders, no commit is needed. - The auxiliary buffer is removed together with the corresponding bootstrap events when they are committed or cleaned up. Does this approach make sense to you? If so, I can clean up the implementation and submit a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
