[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819813#comment-16819813
]
Yingjie Cao commented on FLINK-12070:
-------------------------------------
[~eaglewatcher] Indeed, disks are usually shared in a cluster, so is memory.
But it is hard to emulate the real production environment, because different
applications may have different resource access pattern. May be running several
different applications simultaneously in the test cluster is a easy way. As for
cache, nearly all free memory can be used as cache in our test environment.
[~StephanEwen] The case that a taskmanager with multiple slots participating in
a high parallelism shuffle will be covered in my test.
> Make blocking result partitions consumable multiple times
> ---------------------------------------------------------
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Till Rohrmann
> Assignee: BoWang
> Priority: Major
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
