Re: Automatic backpressure detection

2021-04-13 Thread Lu Niu
Cool. Thanks! Best Lu On Mon, Apr 12, 2021 at 11:27 PM Piotr Nowojski wrote: > Hi, > > Yes. Back-pressure from AsyncOperator should be correctly reported via > isBackPressured, backPressuredMsPerSecond metrics and by extension in the > WebUI from 1.13. > > Piotre > > pon., 12 kwi 2021 o 23:17 L

Re: Automatic backpressure detection

2021-04-12 Thread Piotr Nowojski
Hi, Yes. Back-pressure from AsyncOperator should be correctly reported via isBackPressured, backPressuredMsPerSecond metrics and by extension in the WebUI from 1.13. Piotre pon., 12 kwi 2021 o 23:17 Lu Niu napisaƂ(a): > Hi, Piotr > > Thanks for your detailed reply! It is mentioned here we cann

Re: Automatic backpressure detection

2021-04-12 Thread Lu Niu
Hi, Piotr Thanks for your detailed reply! It is mentioned here we cannot observe backpressure generated from AsyncOperator in Flink UI in 1.9.1. Is it fixed in the latest version? Thank you! http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Async-Function-Not-Generating-Backpres

Re: Automatic backpressure detection

2021-04-06 Thread Piotr Nowojski
Hi, Yes, you can use `isBackPressured` to monitor a task's back-pressure. However keep in mind: a) You are going to miss some nice way to visualize this information, which is present in 1.13's WebUI. b) `isBackPressured` is a sampling based metric. If your job has varying load, for example all win

Re: Automatic backpressure detection

2021-04-06 Thread Lu Niu
Hi, Piotr Thanks for replying! We don't have a plan to upgrade to 1.13 in short term. We are using flink 1.11 and I notice there is a metric called isBackpressured. Is that enough to solve 1? If not, would backporting patches regarding backPressuredTimeMsPerSecond, busyTimeMsPerSecond and idleTim

Re: Automatic backpressure detection

2021-04-06 Thread Piotr Nowojski
Hi, Lately we overhauled the backpressure detection [1] and a screenshot preview of those efforts is attached here [2]. I encourage you to check the 1.13 RC0 build and how the current mechanism works for you [3]. To support those WebUI changes we have added a couple of new metrics: backPressuredTi