Re: YARN parallelism vs Config

2019-07-23 Thread Till Rohrmann
Hi Dominik, the yarn-properties-USER file is mainly a legacy artifact which served the following purpose: Back in the days when starting a Yarn session cluster, one needed to specify the number of task managers with which to start the cluster. This number could not be changed. If now a user wanted

Re: YARN parallelism vs Config

2019-07-22 Thread Jeff Zhang
Hi Dom. I believe this is for per job scenario. That means you create flink cluster in yarn cluster when you submitting your flink. Since this is is per job scenario (there's only one job in this flink cluster), so that it makes more sense to set up the parallelism to be workers * number of slots

YARN parallelism vs Config

2019-07-19 Thread Dominik WosiƄski
Hey, I was wondering about the relation between the parallelism set by YARN in Yarn properties file. Currently, as far as I know there is only one execution of `writeYarnPropertiesFIle` method and it sets the parallelism in the YARN properties to the number of workers * number of slots per worker.