[
https://issues.apache.org/jira/browse/FLINK-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15926090#comment-15926090
]
Luke Hutchison commented on FLINK-6057:
---------------------------------------
Yes, thanks, sorry for the dup.
> Better default needed for num network buffers
> ---------------------------------------------
>
> Key: FLINK-6057
> URL: https://issues.apache.org/jira/browse/FLINK-6057
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.0
> Reporter: Luke Hutchison
>
> Using the default environment,
> {code}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> {code}
> my code will sometimes fail with an error that Flink ran out of network
> buffers. To fix this, I have to do:
> {code}
> int numTasks = Runtime.getRuntime().availableProcessors();
> config.setInteger(ConfigConstants.DEFAULT_PARALLELISM_KEY, numTasks);
> config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, numTasks);
> config.setInteger(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY,
> numTasks * 2048);
> {code}
> The default value of 2048 fails when I increase the degree of parallelism for
> a large Flink pipeline (hence the fix to set the number of buffers to
> numTasks * 2048).
> This is particularly problematic because a pipeline can work fine on one
> machine, and when you start the pipeline on a machine with more cores, it can
> fail.
> The default execution environment should pick a saner default based on the
> level of parallelism (or whatever is needed to ensure that the number of
> network buffers is not going to be exceeded for a given execution
> environment).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
