Hi Averell, It is up to the YARN scheduler on which hosts the containers are started.
What Flink version are you using? I assume you are using 1.4 or earlier because you are specifying a fixed number of TMs. If you launch Flink with -yn 2, you should be only seeing 2 TMs in total (not 4). Are you starting two clusters? Beginning with Flink 1.5, -yn is obsolete because resources are acquired dynamically, and it is not well-defined in what order TM slots are exhausted [1]. Best, Gary [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-5-job-distribution-over-cluster-nodes-td23364.html On Wed, Sep 26, 2018 at 9:25 AM Averell <lvhu...@gmail.com> wrote: > Thank you Gary. > Regarding your previous suggestion to to change the configuration regarding > to the number of vcores on the EMR master node, I tried and found one > funny/bad behaviour as following: > * hardware onfiguration: master node: 4vcores + 8GB ram, 2x executors with > 16vcores + 32GB ram each. > * Flink launch parameters: -yn 2 -ys 16 -ytm 4g... > 4 TMs were created, with 2 of them were used (0 free slots) and two others > not used (16 free slots). The bad thing is most of the time 2 free TMs are > on a same machine, and two occupied ones are on the other machine. > If I dont change the Hadoop configurations then still 4 TMs created, but > the > occupied ones are always on two different servers. > > I'm not sure whether that's EMR's issue, or YARN's or Flink's. > > Thanks and regards, > Averell > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
