Re: [External] Measuring Kafka consumer lag

2020-06-16 Thread Theo Diefenthal
t; An: "Robert Metzger" , "user" Gesendet: Dienstag, 16. Juni 2020 02:52:16 Betreff: Re: [External] Measuring Kafka consumer lag Thanks Robert. Yes we monitor many of the Flink internal metric, which is why I was surprised that we were unable to notice the warning signs bef

Re: [External] Measuring Kafka consumer lag

2020-06-15 Thread Padarn Wilson
Thanks Robert. Yes we monitor many of the Flink internal metric, which is why I was surprised that we were unable to notice the warning signs before our consumers notified us. It would be nice to measure the topic vs consumer group offset of the flink consumer. On Tue, Jun 16, 2020 at 1:57 AM Ro

[External] Measuring Kafka consumer lag

2020-06-13 Thread Padarn Wilson
Hi all, I'm looking for some advice on how other people measure consumer lag for Kafka consumers. Recently we had an application that looked like it was performing identically to before, but all of a sudden the throughput of the job decreased dramatically. However it was not clear from our Flink m