[
https://issues.apache.org/jira/browse/FLINK-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233501#comment-15233501
]
Ufuk Celebi commented on FLINK-1404:
------------------------------------
Yes and no.
This applies to any intermediate result, which holds references to buffers from
the network buffer pool. If we implement some kind of caching (I think as part
of the super issue FLINK-2250), we need a way to trigger the release of these
buffers. This can mean different things for different types of intermediate
results, e.g. just release the result, spill it to disk to allow back tracking,
etc.
Are you planning to work on this part of Flink? :-)
> Add support to cache intermediate results
> -----------------------------------------
>
> Key: FLINK-1404
> URL: https://issues.apache.org/jira/browse/FLINK-1404
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Runtime
> Reporter: Ufuk Celebi
> Assignee: Maximilian Michels
>
> With blocking intermediate results (FLINK-1350) and proper partition state
> management (FLINK-1359) it is necessary to allow the network buffer pool to
> request eviction of historic intermediate results when not enough buffers are
> available. With the currently available pipelined intermediate partitions
> this is not an issue, because buffer pools can be released as soon as a
> partition is consumed.
> We need to be able to trigger the recycling of buffers held by historic
> intermediate results when not enough buffers are available for new local
> pools.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
