Hello, Chris. Thanks for the feedback.
> Have you seen the consumer-side lag metrics [1]? "records-lag», Yes, I’m aware of these metrics. > If so, I'd be curious to know what the motivation for duplicating existing > client metrics onto brokers would be? It can be a complex task to setup and access monitoring data for all Consumers. Clients can be new, experimental and not integrated into company monitoring solution. Instances can come and go; Clients can change addresses, etc. based on some circumstances not related to Kafka. I think it will be useful to have per partition consumer offset metrics on broker side. It allows to Kafka administrator collect monitoring data in one place. Moreover, these data available on the broker, already. All we need is to expose them. Makes sense for you? > 16 февр. 2022 г., в 17:01, Chris Egerton <chr...@confluent.io.INVALID> > написал(а): > > Hi Nikolay, > > Have you seen the consumer-side lag metrics [1]? "records-lag", > "records-lag-avg", "records-lag-max" all give lag stats on a > per-topic-partition basis. > > If so, I'd be curious to know what the motivation for duplicating existing > client metrics onto brokers would be? > > [1] https://kafka.apache.org/31/documentation.html#consumer_fetch_monitoring > > Cheers, > > Chris > > On Wed, Feb 16, 2022 at 4:38 AM Nikolay Izhikov <nizhi...@apache.org> wrote: > >> Hello, Kafka team. >> >> When running in production the common user question is «How big lag >> between producer and consumer?». >> We have a `kafka-consumer-groups.sh` tool and >> `AdminClient#getListConsumerGroupOffsetsCall` to answer the question. >> >> Even detailed guides on how to calculate *consumer lag* with built-in >> Kafka tools, exists. [1] >> >> Obviously, approach with tool or AdminClient requires additional coding >> and setup which can be inconvenient. >> >> I think Kafka should provide per partition consumer offset metric. >> It will simplify running Kafka deployment and monitoring in production. >> >> Looked in `GroupMetadataManager.scala` and think it possible to add those >> metrics. >> >> What do you think? >> Do we need this metrics on Kafka broker? >> >> [1] https://www.baeldung.com/java-kafka-consumer-lag
