Ufuk Celebi created FLINK-1709:
----------------------------------
Summary: Add support for programs with higher-than-slot parallelism
Key: FLINK-1709
URL: https://issues.apache.org/jira/browse/FLINK-1709
Project: Flink
Issue Type: Improvement
Affects Versions: master
Reporter: Ufuk Celebi
Currently, we can't run programs with higher parallelism than available slots.
For example, if we currently have a map-reduce program and 4 task slots
configured (e.g. 2 task managers with 2 slots per task manager), the map and
reduce tasks will be scheduled with pipelined results and the same parallelism
in shared slots. Setting the parallelism to more than available slots will
result in a NoResourcesAvailableException.
As a first step to support these kinds of programs, we can add initial support
for this when running in batch mode (after
https://github.com/apache/flink/pull/471 is merged).
This is easier than the original pipelined scenario, because the map tasks can
be deployed after each other to produce the blocking result. The blocking
result can then be consumed after all map tasks produced their result. The
mechanism in #471 to deploy result receivers can be used for this and should
not need any modifications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
