Hi Max, the community is currently working on Flink's scheduler component [1]. One of the things we want to enable in the future is bulk scheduling. With this, it should also be possible to add strategies how to distribute tasks across multiple TMs (spreading vs. co-locating).
In general, I'm not 100% sure whether spreading out tasks is always the best strategy. Especially if you have a network heavy job co-locating tasks on the same TM could have benefits over spreading the tasks out. [1] https://issues.apache.org/jira/browse/FLINK-10429 Cheers, Till On Thu, Oct 11, 2018 at 8:16 PM Maximilian Michels <m...@apache.org> wrote: > Hi everyone, > > I've recently come across a cluster scheduling problem users are facing. > Clusters where TaskManagers have more slots than the parallelism > (#tm_slots > job_parallelism), tend to schedule all job tasks on a > single TaskManager. > > This is not good for spreading load and has been discussed in FLINK-1003 > [1] and the other duplicate JIRA issues. > > I know that this is not really an issue if the cluster is created > exclusively for the Job, or if the number of slots per Taskmanager is > smaller than the parallelism. However, this seems like a rather easy > improvement to the Scheduler which would have a huge impact on performance. > > On the JIRA issue page it has been mentioned that this was put on hold > to work on dynamic scaling first. > > Now that the basic building blocks for dynamic scaling are in place, do > you think it would be possible to tackle FLINK-1003? > > Thanks, > Max > > > [1] https://issues.apache.org/jira/browse/FLINK-1003 >
