Thanks Robert. Yes we monitor many of the Flink internal metric, which is why I was surprised that we were unable to notice the warning signs before our consumers notified us.
It would be nice to measure the topic vs consumer group offset of the flink consumer. On Tue, Jun 16, 2020 at 1:57 AM Robert Metzger <rmetz...@apache.org> wrote: > Hi Padarn, > I usually recommend the approach you described: accessing/monitoring the > lag via Flink's metrics system. Sometimes it also makes sense to consider > application level metrics. > I checked Youtube for past Flink Forward talks, but I couldn't find a > video. I'm sure there were users talking about best practices for > monitoring Flink in the past ... > > Best, > Robert > > On Sun, Jun 14, 2020 at 5:47 AM Padarn Wilson <padarn.wil...@grab.com> > wrote: > >> Hi all, >> >> I'm looking for some advice on how other people measure consumer lag for >> Kafka consumers. Recently we had an application that looked like it was >> performing identically to before, but all of a sudden the throughput of the >> job decreased dramatically. However it was not clear from our Flink >> metrics, only from the lag in time vs watermark time that our consumers >> were measuring. >> >> How do people approach measuring this? >> >> Thanks, >> Padarn >> >> >> By communicating with Grab Inc and/or its subsidiaries, associate >> companies and jointly controlled entities (“Grab Group”), you are deemed to >> have consented to the processing of your personal data as set out in the >> Privacy Notice which can be viewed at https://grab.com/privacy/ >> >> This email contains confidential information and is only for the intended >> recipient(s). If you are not the intended recipient(s), please do not >> disseminate, distribute or copy this email Please notify Grab Group >> immediately if you have received this by mistake and delete this email from >> your system. Email transmission cannot be guaranteed to be secure or >> error-free as any information therein could be intercepted, corrupted, >> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do >> not accept liability for any errors or omissions in the contents of this >> email arises as a result of email transmission. All intellectual property >> rights in this email and attachments therein shall remain vested in Grab >> Group, unless otherwise provided by law. >> > -- By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/ <https://grab.com/privacy/> This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.
