cshuo commented on code in PR #19951:
URL: https://github.com/apache/hudi/pull/19951#discussion_r4022049033
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -435,6 +439,15 @@ private CompletableFuture<CoordinationResponse>
handleInFlightInstantsRequest(Co
return
CompletableFuture.completedFuture(CoordinationResponseSerDe.wrap(coordinationResponse));
}
+ private CompletableFuture<CoordinationResponse>
handlePendingBucketFileIdsRequest(Correspondent.PendingBucketFileIdsRequest
request) {
Review Comment:
@zhaoyudi-creator Thanks for the update. Querying pending fileIds before
loading a fresh committed view addresses the original missing-fileId window,
assuming the relevant recovery metadata is already registered. I still have two
concerns:
1. Blocking the writer’s write flow. The lookup shares the single-threaded
executor with recommit. If recommit is running, the request waits behind it,
and the writer synchronously waiting for the response cannot continue
processing records. This effectively couples partition bootstrap to commit
latency.
2. RPC timeouts during slow recommits. This request uses Flink’s
coordination RPC, whose ask timeout defaults to [10
seconds](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/RpcOptions.java).
A slow recommit can exhaust that timeout before the lookup executes, causing a
task failure rather than just delaying bootstrap.
[#19902](https://github.com/apache/hudi/issues/19902) reports a similar failure
mode for instant-time requests waiting on coordinator-side work.
cc @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
