Hi experts I am running flink job on YARN in job cluster mode, the job is divided into 2 tasks, the following are some configs of the job: parallelism.default => 16 taskmanager.numberOfTaskSlots => 8 -yn => 2
when the program starts, I found that the count of task managers is not set immediately, but first expand then converge, I record the number during the process: Task Managers Task Slots Available Task Slots 1. 14 104 88 2. 15 120 104 3. 16 128 112 4. 6 48 32 5. 3 24 8 6. 2 16 0 The final state is correct. There are 2 tasks, 32 subtask in total, due to slot sharing, only 16 slots are enough, the number of task slots per TM are 8, so 2 TMs are needed. I have the following question: Because I specify yn=2, why does not directly allocate 2 TMs, but goes the expand then converge process? Why does it apply 16 task managers at most? If it is not a must, how to avoid it? Thanks a lot! Best Henry