As far as I know, Flink does not have any requirements on how the TaskManagers are distributed across physical machines. So I think it really depends on the scheduling policy of the Mesos cluster. I'm not an expert on Mesos, so correct me if I was wrong.
Thank you~ Xintong Song On Fri, May 24, 2019 at 4:18 PM black chase <chaseandbl...@gmail.com> wrote: > Hi Song, > Thank you for the clarification. > Now I know TaskManagers are automatically allocated. Yet, I am still not > very clear how the TMs are allocated. > I'm guessing the allocation process would be: > On the job side, I have a job with each operator parallelism=5. Since one > TaskManager has one slot. It means this job would need 5 TaskManagers > becase one pipeline would need one slot according to the task scheduling > policy. > On the Mesos side, let's say currently there are no available > TaskManagers. Then Mesos would spawn 5 new TaskManagers for this job. Is it? > If what I am guessing is right, then let's say we have 4 physical > computing nodes for the Flink TaskManagers. How would Mesos place the 5 new > TaskManagers into the 4 physical computing nodes? Is it juts a Round-Robin > fashion? > Kind regards > Chase > > > > On Fri, May 24, 2019 at 4:10 AM Xintong Song <tonysong...@gmail.com> > wrote: > >> Hi black, >> >> If you are running Flink on Yarn or Mesos, Flink will automatically >> allocate resource and launch new TaskManagers as needed. >> >> If you are using Flink standalone mode, then the easiest way is to enable >> slot sharing and set all the vertices into the same group (which is by >> default). In that way, the total slots (or number of TaskManagers if you >> config on slot for each TaskManager) needed for running the job would be >> the maximum parallelism of the job graph vertices. Further information on >> slot sharing could be found here >> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/runtime.html#task-slots-and-resources> >> . >> >> Thank you~ >> >> Xintong Song >> >> >> >> On Thu, May 23, 2019 at 11:49 PM black chase <chaseandbl...@gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> I am redesigning the scheduler of the JobManager to place tasks of a job >>> across TaskManagers accroding to a scheduling policy. >>> >>> I am reading the Flip-6 proposal and found that the common case is "one >>> TaskManager launchs one slot", and "one Flink cluster serves one job". But >>> I did not find how many TaskManagers to launch in a computing node. Is >>> there any common practice for this ? >>> >>> -- >>> Best Regards! >>> Pengcheng Duan >>> >> > > -- > Best Regards! >
