zhaoyudi-creator commented on code in PR #19951:
URL: https://github.com/apache/hudi/pull/19951#discussion_r4025790213
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -435,6 +439,15 @@ private CompletableFuture<CoordinationResponse>
handleInFlightInstantsRequest(Co
return
CompletableFuture.completedFuture(CoordinationResponseSerDe.wrap(coordinationResponse));
}
+ private CompletableFuture<CoordinationResponse>
handlePendingBucketFileIdsRequest(Correspondent.PendingBucketFileIdsRequest
request) {
Review Comment:
@danny0405 @cshuo Thanks for the careful review — both concerns are valid,
and following them through leads to a conclusion worth stating explicitly.
On restore the coordinator recommits in two places:
Path 1 — restoreEvents: executed synchronously inside start() /
resetToCheckpoint(). Subtasks are deployed only after it returns successfully,
which means Path 1's recommit always completes before bootstrapIndexIfNeed runs.
Path 2 — recommitInstant inside handleBootstrapEvent: dispatched via
executor.execute, where buffer population and recommit run in the same task and
share the single-threaded executor with the lookup. This is exactly where it
can go wrong: a slow recommit blocks the lookup and can exhaust the 10s ask
timeout.
Path 2 also exposes a correctness window under rescale. Under rescale the
mapping changes, so the task that owns the bucket and issues the lookup is
generally not the task that resends fileId-A. Two different subtasks have no
ordering guarantee toward the coordinator.
Conclusion: a lookup that does not sit on the commit path cannot guarantee
the absence of duplicate fileIds — blocking and correctness are two ends of the
same trade-off. That leaves only two ways forward:
1. Wait for the recommit to succeed — have the lookup explicitly wait for
the instant to be recommitted (accepting the coupling with commit latency);
2. Deterministic fileId — per-bucket, though the insert-overwrite scenario
needs to be considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
