cshuo commented on code in PR #19951:
URL: https://github.com/apache/hudi/pull/19951#discussion_r4015765476
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -435,6 +439,15 @@ private CompletableFuture<CoordinationResponse>
handleInFlightInstantsRequest(Co
return
CompletableFuture.completedFuture(CoordinationResponseSerDe.wrap(coordinationResponse));
}
+ private CompletableFuture<CoordinationResponse>
handlePendingBucketFileIdsRequest(Correspondent.PendingBucketFileIdsRequest
request) {
Review Comment:
Preserve recovery fileIds after recommit clears the pending buffer
The pending-only lookup still leaves the original race open: the writer can
read the committed view before recommit finishes, then execute this lookup
after recommit has completed and cleared the event buffer. Both reads miss the
original fileId, causing the writer to generate another fileId for the same
bucket. Serializing this lookup with recommit on the same executor does not
prevent that sequence.
One possible solution is: we maintain a separate partition-> bucketId ->
fileId recovery mapping, populated from restored coordinator state and incoming
bootstrap events, and retain it for the coordinator’s lifetime? Bootstrap could
merge this mapping with the committed view, giving committed entries
precedence. This would preserve the original fileId across pending-buffer
cleanup without waiting for recommit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
