Re: flink job : TPS drops from 400 to 30 TPS

2021-09-27 Thread JING ZHANG
Hi, About cpu cost, there are several methods: 1. Flink builtin metric: `Status.JVM.CPU.Load` [1] 2. Use `top` command on the target machine which deploys a suspect TaskManager 3. You could use flame graph to do deeper profiler of a JVM [2]. ... About RPC response, I'm not an expert on HBase, I'm

Re: flink job : TPS drops from 400 to 30 TPS

2021-09-27 Thread Ragini Manjaiah
please let me know how to check Does RPC response and CPU cost On Mon, Sep 27, 2021 at 1:19 PM JING ZHANG wrote: > Hi, > Since there is not enough information, you could first check the back > pressure status of the job [1], find the task which caused the back > pressure. > Then try to find out

Re: flink job : TPS drops from 400 to 30 TPS

2021-09-27 Thread JING ZHANG
Hi, Since there is not enough information, you could first check the back pressure status of the job [1], find the task which caused the back pressure. Then try to find out why the task processed data slowly, there are many reasons, for example the following reasons: (1) Does data skew exist, which

flink job : TPS drops from 400 to 30 TPS

2021-09-26 Thread Ragini Manjaiah
Hi , I have a flink real time job which processes user records via topic and also reading data from hbase acting as a look table . If the look table does not contain required metadata then it queries the external db via api . First 1to 2 hours it works fine without issues, later it drops down dras