Hi, Thanks for replying! Basically I want to limit cpu usage so that different application don't affect each other. What's current best practice? Looks `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true` is one way. How to set how many cpu resources to use? is it "yarn.containers.vcores" ?
it should be -ys not -yn in original post, sorry for the typo. Best Lu On Wed, Nov 6, 2019 at 1:41 AM Yang Wang <danrtsey...@gmail.com> wrote: > If you want to limit the TaskManager container cpu usage, it is based on > your yarn cluster configuration. > By default, yarn only uses cpu share. You need to set > `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage=true` > in yarn-site.xml of all yarn node managers. > > > Best, > Yang > > Victor Wong <jiasheng.w...@outlook.com> 于2019年11月6日周三 下午5:02写道： > >> Hi Lu, >> >> >> >> You can check out which operator thread causes the high CPU usage, and >> set a unique slot sharing group name [1] to it to prevent too many operator >> threads running in the same TM. >> >> Hope this will be helpful😊 >> >> >> >> [1]. >> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#task-chaining-and-resource-groups >> >> >> >> Best, >> >> Victor >> >> >> >> *From: *Vino Yang <yanghua1...@gmail.com> >> *Date: *Wednesday, 6 November 2019 at 4:26 PM >> *To: *Lu Niu <qqib...@gmail.com> >> *Cc: *user <user@flink.apache.org> >> *Subject: *Re: Limit max cpu usage per TaskManager >> >> >> >> Hi Lu, >> >> >> >> When using Flink on YARN, it will rely on YARN's resource management >> capabilities, and Flink cannot currently limit CPU usage. >> >> Also, what version of Flink do you use? As far as I know, since Flink >> 1.8, the -yn parameter will not work. >> >> >> >> Best, >> >> Vino >> >> >> >> Lu Niu <qqib...@gmail.com> 于2019年11月6日周三 下午1:29写道： >> >> Hi, >> >> >> >> When run flink application in yarn mode, is there a way to limit maximum >> cpu usage per TaskManager? >> >> >> >> I tried this application with just source and sink operator. >> parallelism of source is 60 and parallelism of sink is 1. When running in >> default config, there are 60 TaskManager assigned. I notice one TaskManager >> process cpu usage could be 200% white the rest below 50%. >> >> >> >> When I set -yn = 2 (default is 1), I notice # of TaskManger dropped down >> to 30. and one TaskManger process cpu usage could be 600% while the rest >> below 50%. >> >> >> >> Tried to set yarn.containers.vcores = 2, all tasks are in start state >> forever, application is not able to turn to running state. >> >> >> >> Best >> >> Lu >> >>
