Deep, 1) Is it a cpu/memory/io intensive job ??
Based on that you could allocate resources. >From the question, if the CPU is not utilised , you could run multiple containers on the same machine(tm) ... Following may not be exact case as yours but to give you an idea. Few months back I have run jobs in emr processing 4-8k per second from kafka with paralleism of 8 doing lightweight transformation where end to end latency was less than a second (10-50ms). I used slots where memory allocated is 4GB and JM memory 1gb. Here multilple containers ran on the same machine and I got cpu usgae upto 50%. Earlier it was in single digits when just single container ran on a machine. Prasanna. On Thu 5 Nov, 2020, 12:40 Satyaa Dixit, <satyaadi...@gmail.com> wrote: > Hi Deep, > > Thanks for bringing this on table, I'm also facing a similar kind of issue > while deploying my flink Job w.r.t resources optimization. > > Hi Team, > > It would be much appreciated if someone helps us here. > > > Regards, > Satya > > On Wed, Nov 4, 2020 at 6:33 PM DEEP NARAYAN Singh <about.d...@gmail.com> > wrote: > > > Hi All, > > > > I am running a flink streaming job in EMR Cluster with parallelism 21 > > having 500 records per second.But still seeing cpu utilization is > > approximate 5-8 percent. > > > > Below is the long running session command in EMR Cluster having 3 > instance > > of type C52xlarge(8vcore, 16 GB memory, AWS resource) > > > > *sudo flink-yarn-session -n 3 -s 7 -jm 4168 -tm 8000 -d* > > > > Anyone can suggest some configuration to maximize the CPU utilization? > > And Also what would be the standard utilization of CPU for flink job in > > order to achieve the minimum latency? > > > > Any leads would be appreciated. > > > > Thanks, > > -Deep > > > > > -- > -------------------------- > Best Regards > Satya Prakash > (M)+91-9845111913 >
