Re: YARN per-job cluster reserves all of remaining memory in YARN

2018-05-13 Thread Till Rohrmann
Hi Dongwon, Fabian is right with his analysis of the problem. Currently, the new ResourceManager implementation will start a new TM per requested slot independent of the number of configured slots. This behavior can cause that the cluster requests too many resources when a job is started. However,

Re: YARN per-job cluster reserves all of remaining memory in YARN

2018-05-07 Thread Fabian Hueske
Hi Dongwon, I see that you are using the latest master (Flink 1.6-SNAPSHOT). This is a known problem in the new FLIP-6 mode. The ResourceManager tries to allocate too many resources, basically on TM per required slot, i.e., it does not take the number of slots per TM into account. The resources ar