Florin, What may be of use is seeing the "consumer lag”. This is the offset position of the latest message consumed by the consumer group, per partition, vs. the last offset of the topic. So if a lag is 3, that means there are 3 unconsumed messages by the consumer group.
I’m unsure where it would be available via JMX but I can inspect it using the following command, running on the Kafka instance: kafka-consumer-groups.sh --bootstrap-server <kafka broker hostname:port> --describe --group <consumer group name> This link has a great description of the command and example output: https://docs.cloudera.com/runtime/7.2.10/kafka-managing/topics/kafka-manage-cli-cgroups.html#pnavId1 <https://docs.cloudera.com/runtime/7.2.10/kafka-managing/topics/kafka-manage-cli-cgroups.html#pnavId1> Perhaps knowing this could point you to the metric you want. Hope it helps! Nhat > On Jan 11, 2022, at 2:26 PM, Florin Andrei <flo...@andrei.myip.org> wrote: > > We have several producers and several consumers connected via Kafka - > basically, it's batch processing jobs that are created on demand, placed on > Kafka in several queues, and the batch processors pick them up via Kafka and > process them one by one. > > I want to visualize and monitor the lengths of these queues in Kafka. The > queue lengths will serve as proxies for the "load" of the overall system. The > more jobs waiting in queues, the more "loaded" the system is. > > Our Kafka is an AWS MSK cluster. I've enabled Prometheus JMX monitoring, and > I'm scraping all metrics every 10 seconds. > > Looking at the metrics, there's nothing that appears immediately obvious in > terms of queue length. Is that parameter exposed as a metric by default? > > If queue length is not exposed by default, what is a good way to collect that > metric? Assume I can write a Python script with any libraries installed, and > I have full access to the Kafka endpoints from it. > > Note: I understand the basic concepts, but I don't have a lot of practical > experience with Kafka (I've only used RabbitMQ before), so apologies if my > vocabulary is not very precise. > > -- > Florin Andrei > https://florin.myip.org/
