Thanks Yangze The reason why I don’t deploying a standalone cluster, it's because there kafka, kudu, hadoop, zookeeper on these machines, maybe currently using the yarn to manage resources is the best choice for me. If Flink can not control how many tm to start , could anyone providing me some best practice for deploying on yarn please? I read the [1] and still don't very clear
[1] https://www.ververica.com/blog/how-to-size-your-apache-flink-cluster-general-guidelines -----邮件原件----- 发件人: Yangze Guo [mailto:karma...@gmail.com] 发送时间: 2020年8月18日 星期二 10:50 收件人: 范超 <fanc...@mgtv.com> 抄送: user (user@flink.apache.org) <user@flink.apache.org> 主题: Re: How to specify the number of TaskManagers in Yarn Cluster using Per-Job Mode Hi, I think that is only related to the Yarn scheduling strategy. AFAIK, Flink could not control it. You could check the RM log to figure out why it did not schedule the containers to all the three machines. BTW, if you have specific requirements to start with all the three machines, how about deploying a standalone cluster instead? Best, Yangze Guo On Tue, Aug 18, 2020 at 10:24 AM 范超 <fanc...@mgtv.com> wrote: > > Thanks Yangze > > All 3 machines NodeManager is started. > > I just don't know why not three machines each running a Flink > TaskManager and how to achieve this > > -----邮件原件----- > 发件人: Yangze Guo [mailto:karma...@gmail.com] > 发送时间: 2020年8月18日 星期二 10:10 > 收件人: 范超 <fanc...@mgtv.com> > 抄送: user (user@flink.apache.org) <user@flink.apache.org> > 主题: Re: How to specify the number of TaskManagers in Yarn Cluster > using Per-Job Mode > > Hi, > > Do you start the NodeManager in all the three machines? If so, could you > check all the NMs correctly connect to the ResourceManager? > > Best, > Yangze Guo > > On Tue, Aug 18, 2020 at 10:01 AM 范超 <fanc...@mgtv.com> wrote: > > > > Hi, Dev and Users > > I’ve 3 machines each one is 8 cores and 16GB memory. > > Following it’s my Resource Manager screenshot the cluster have 36GB total. > > I specify the paralism to 3 or even up to 12, But the task manager is > > always running on two nodes not all three machine, the third node does not > > start the task manager. > > I tried set the –p –tm –jm parameters, but it always the same, only > > different is more container on the two maching but not all three machine > > start the task manager. > > My question is how to set the cli parameter to start all of my three > > machine (all task manager start on 3 machines) > > > > Thanks a lot > > [cid:image001.png@01D67546.62291B70] > > > > > > Chao fan > >
