[
https://issues.apache.org/jira/browse/FLINK-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154042#comment-16154042
]
ASF GitHub Bot commented on FLINK-7582:
---------------------------------------
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/4645#discussion_r137065666
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
@@ -159,24 +155,17 @@ public void shutdown(Time timeout) {
}
private <P extends ResponseBody> CompletableFuture<P>
submitRequest(String targetAddress, int targetPort, FullHttpRequest
httpRequest, Class<P> responseClass) {
- return CompletableFuture.supplyAsync(() ->
bootstrap.connect(targetAddress, targetPort), executor)
- .thenApply((channel) -> {
- try {
- return channel.sync();
- } catch (InterruptedException e) {
- throw new FlinkRuntimeException(e);
- }
- })
- .thenApply((ChannelFuture::channel))
- .thenCompose(channel -> {
- ClientHandler handler =
channel.pipeline().get(ClientHandler.class);
- CompletableFuture<JsonResponse> future =
handler.getJsonFuture();
- channel.writeAndFlush(httpRequest);
- return future;
- }).thenComposeAsync(
- (JsonResponse rawResponse) ->
parseResponse(rawResponse, responseClass),
- executor
- );
+ ChannelFuture connect = bootstrap.connect(targetAddress,
targetPort);
+ Channel channel;
--- End diff --
Technically yes, this alone would fix the problem at hand.
The sending part was modified as a band-aid to alleviate issues that can
occur under heavy load since we don't have any throttling mechanism for
accepting requests. For example, in one of our tests we had thousands of
futures suddenly fail because all ports were in use, not to mention that you
can allocate a virtually infinite number of futures (way faster than we could
ever process them).
I couldn't quickly come up with a good scheme to throttle requests though.
Suggestions are welcome ;)
> Rest client may buffer responses indefinitely under heavy laod
> --------------------------------------------------------------
>
> Key: FLINK-7582
> URL: https://issues.apache.org/jira/browse/FLINK-7582
> Project: Flink
> Issue Type: Bug
> Components: REST
> Affects Versions: 1.4.0
> Reporter: Chesnay Schepler
> Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> The RestClient uses an executor for sending requests and parsing responses.
> Under heavy load, i.e. lots of requests being sent, the executor may be used
> exclusively for sending requests. The responses that are received by the
> netty threads are thus never parsed and are buffered in memory, until either
> requests stop coming in or all memory is used up.
> We should let the netty receiver thread do the parsing as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
