Yes true. I am trying to figure out how the TaskManagers are distributed across physical machines by Mesos and YARN. Maybe I shoud start a new thread for help. Thank you Song Best, Pengcheng
On Fri, May 24, 2019 at 10:52 AM Xintong Song <tonysong...@gmail.com> wrote: > As far as I know, Flink does not have any requirements on how the > TaskManagers are distributed across physical machines. So I think it really > depends on the scheduling policy of the Mesos cluster. I'm not an expert on > Mesos, so correct me if I was wrong. > > Thank you~ > > Xintong Song > > > > On Fri, May 24, 2019 at 4:18 PM black chase <chaseandbl...@gmail.com> > wrote: > >> Hi Song, >> Thank you for the clarification. >> Now I know TaskManagers are automatically allocated. Yet, I am still not >> very clear how the TMs are allocated. >> I'm guessing the allocation process would be: >> On the job side, I have a job with each operator parallelism=5. Since one >> TaskManager has one slot. It means this job would need 5 TaskManagers >> becase one pipeline would need one slot according to the task scheduling >> policy. >> On the Mesos side, let's say currently there are no available >> TaskManagers. Then Mesos would spawn 5 new TaskManagers for this job. Is it? >> If what I am guessing is right, then let's say we have 4 physical >> computing nodes for the Flink TaskManagers. How would Mesos place the 5 new >> TaskManagers into the 4 physical computing nodes? Is it juts a Round-Robin >> fashion? >> Kind regards >> Chase >> >> >> >> On Fri, May 24, 2019 at 4:10 AM Xintong Song <tonysong...@gmail.com> >> wrote: >> >>> Hi black, >>> >>> If you are running Flink on Yarn or Mesos, Flink will automatically >>> allocate resource and launch new TaskManagers as needed. >>> >>> If you are using Flink standalone mode, then the easiest way is to >>> enable slot sharing and set all the vertices into the same group (which is >>> by default). In that way, the total slots (or number of TaskManagers if you >>> config on slot for each TaskManager) needed for running the job would be >>> the maximum parallelism of the job graph vertices. Further information on >>> slot sharing could be found here >>> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/runtime.html#task-slots-and-resources> >>> . >>> >>> Thank you~ >>> >>> Xintong Song >>> >>> >>> >>> On Thu, May 23, 2019 at 11:49 PM black chase <chaseandbl...@gmail.com> >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> I am redesigning the scheduler of the JobManager to place tasks of a >>>> job across TaskManagers accroding to a scheduling policy. >>>> >>>> I am reading the Flip-6 proposal and found that the common case is "one >>>> TaskManager launchs one slot", and "one Flink cluster serves one job". But >>>> I did not find how many TaskManagers to launch in a computing node. Is >>>> there any common practice for this ? >>>> >>>> -- >>>> Best Regards! >>>> Pengcheng Duan >>>> >>> >> >> -- >> Best Regards! >> > -- Best Regards!
