[
https://issues.apache.org/jira/browse/FLINK-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15445435#comment-15445435
]
Till Rohrmann commented on FLINK-4419:
--------------------------------------
At the moment, the blocking results are kept in managed memory and then spilled
to local files if I'm not mistaken. These results will be released once they
are consumed. So at the moment, the data is lost if a TM dies. Adding DFS
backed output/input channel will resolve this problem.
> Batch improvement for supporting dfs as a ResultPartitionType
> -------------------------------------------------------------
>
> Key: FLINK-4419
> URL: https://issues.apache.org/jira/browse/FLINK-4419
> Project: Flink
> Issue Type: Improvement
> Components: Batch Connectors and Input/Output Formats
> Reporter: shuai.xu
> Assignee: shuai.xu
>
> This is the root issue to track a improvement for batch, which will enable
> dfs as a ResultPartitionType, so that upstream node can exist totally after
> finished and need not be restarted if downstream nodes fail.
> Full design is shown in
> (https://docs.google.com/document/d/15HtCtc9Gk8SyHsAezM7Od1opAHgnxLeHm3VX7A8fa-4/edit#).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
