Hi Zheng, To divide managed memory for operators[1], we need to consider which tasks will run in the same slot. In batch jobs, vertices in different regions may not run at the same time. If we put them in the same slot sharing group, running tasks may run slower with less managed memory, while managed memory reserved for tasks that are not running yet will be wasted.
More slots can be requested as a result. However, it's not necessary to add more containers. One slot will serve fewer tasks, this means you can decrease the slot size (via increase "taskmanager.numberOfTaskSlots") so that the previous number of containers can be enough. This is because you are still running that many tasks at the same time, although they are spread into more slots. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management Thanks, Zhu zheng faaron <faaronzh...@gmail.com> 于2020年9月16日周三 上午10:58写道： > Hi All, > > I find we set AllVerticesInSameSlotSharingGroupByDefault to false in > flink 1.10. It will make batch job request lots of containers. I'm not sure > why we set it to false directly. I try to set it to true and find the batch > job can run correctly with a small amount containers. Why don't we add a > configuration to let user configure it? > > Best, > Faaron Zheng >
