[
https://issues.apache.org/jira/browse/FLINK-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863855#comment-16863855
]
Stephan Ewen commented on FLINK-12070:
--------------------------------------
Is a 5GB network buffers setup common? I think by default settings, Flink never
takes more than 10% (max 1GB) in network buffers.
In such a setup (lots of JVM process memory dedicated to network buffers,
little left outside the JVM process), that naturally favors the previous
implementation over the new implementation.
How would a setup with a smaller JVM process (fewer network buffers) and 4-5GB
memory outside the JVM perform?
The new implementation stared with the assumption of a moderate amount of
network buffer memory, which is common for streaming implementations, so that
we don't have to apply different configurations for batch setups.
If there is an inherent disagreement with that assumption, then we need to
clarify what we want to assume.
Otherwise we work on contradicting assumptions.
> Make blocking result partitions consumable multiple times
> ---------------------------------------------------------
>
> Key: FLINK-12070
> URL: https://issues.apache.org/jira/browse/FLINK-12070
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Till Rohrmann
> Assignee: Stephan Ewen
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Attachments: image-2019-04-18-17-38-24-949.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In order to avoid writing produced results multiple times for multiple
> consumers and in order to speed up batch recoveries, we should make the
> blocking result partitions to be consumable multiple times. At the moment a
> blocking result partition will be released once the consumers has processed
> all data. Instead the result partition should be released once the next
> blocking result has been produced and all consumers of a blocking result
> partition have terminated. Moreover, blocking results should not hold on slot
> resources like network buffers or memory as it is currently the case with
> {{SpillableSubpartitions}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
