Hi, Piotr Thanks for replying!
We don't have a plan to upgrade to 1.13 in short term. We are using flink 1.11 and I notice there is a metric called isBackpressured. Is that enough to solve 1? If not, would backporting patches regarding backPressuredTimeMsPerSecond, busyTimeMsPerSecond and idleTimeMsPerSecond work? And do you have an estimate of how difficult it is? Best Lu On Tue, Apr 6, 2021 at 12:18 AM Piotr Nowojski <pnowoj...@apache.org> wrote: > Hi, > > Lately we overhauled the backpressure detection [1] and a screenshot > preview of those efforts is attached here [2]. I encourage you to check the > 1.13 RC0 build and how the current mechanism works for you [3]. To support > those WebUI changes we have added a couple of new metrics: > backPressuredTimeMsPerSecond, busyTimeMsPerSecond and idleTimeMsPerSecond. > > 1. I believe that solves 1. > 2. This still requires a bit of manual investigation. Once you locate > backpressuring task, you can check the detail subtask stats to check if all > parallel instances are uniformly backpressured/busy or not. If you would > like to add a hint "it looks like you have a data skew in Task XYZ ", that > I believe could be added to the WebUI. > 3. The tricky part is how to display this kind of information. Currently I > would recommend just export/report > backPressuredTimeMsPerSecond, busyTimeMsPerSecond and idleTimeMsPerSecond > metrics for every task to an external system and display them for example > in Graphana. > > The blog post you are referencing is quite outdated, especially with those > new changes from 1.13. I'm hoping to write a new one pretty soon. > > Piotrek > > [1] https://issues.apache.org/jira/browse/FLINK-14712 > [2] > > https://issues.apache.org/jira/browse/FLINK-14814?focusedCommentId=17256926&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17256926 > [3] > > http://mail-archives.apache.org/mod_mbox/flink-user/202104.mbox/%3c1d2412ce-d4d0-ed50-6181-1b610e16d...@apache.org%3E > > pon., 5 kwi 2021 o 23:20 Lu Niu <qqib...@gmail.com> napisał(a): > > > Hi, Flink dev > > > > Lately, we want to develop some tools to: > > 1. show backpressure operator without manual operation > > 2. Provide suggestions to mitigate back pressure after checking data > skew, > > external service RPC etc. > > 3. Show back pressure history > > > > Could anyone share their experience with such tooling? > > Also, I notice backpressure monitoring and detection is mentioned across > > multiple places. Could someone help to explain how these connect to each > > other? Maybe some of them are outdated? Thanks! > > > > 1. The official doc introduces monitoring back pressure through web UI. > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/monitoring/back_pressure.html > > 2. In https://flink.apache.org/2019/07/23/flink-network-stack-2.html, it > > says outPoolUsage, inPoolUsage metrics can be used to determine back > > pressure. > > 3. Latest flink version introduces metrics called “isBackPressured" But I > > didn't find related documentation on usage. > > > > Best > > Lu > > >
