Hello team, JMX might not help as Kafka JMX doesn't provide Kafka.consumer Mbeans. However, this Mbean is provided by consumer application. So I am trying to fetch JMX data from the consumer application with the help of Zabbix, but I'm unable to fetch proper data.
JMX keys tried - - jmx.discovery[beans,"kafka.consumer:type=consumer-fetch-manager-metrics,client-id=consumer-cg-campaign-customer-campaign-customer-consumer-group*"] - jmx.get[beans,"kafka.consumer:type=consumer-fetch-manager-metrics,client-id=consumer-cg-campaign-customer-campaign-customer-consumer-group*"] Can anyone provide any other tool recommendations (tool should work without zookeeper) or commands which can provide lag for any historical instance? [image: https://exotel.com/] Akshay Kumar Member of Technical Staff - 2 | Exotel <https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature> [image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter] <https://twitter.com/Exotel> [image: Linkedin] <https://www.linkedin.com/company/exotel-techcom-private-limited/> Ph: +918556063696 Latest from the Exotel Blog <https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/> On Wed, May 10, 2023 at 12:28 AM Santhosh Kumar <greetsanth...@gmail.com> wrote: > Hi Akshay > > Burrow is a popular open-source tool for monitoring the consumer lag in > Apache Kafka. However, Burrow depends on ZooKeeper to access the Kafka > metadata and track the consumer lag. Therefore, it is not possible to use > Burrow without ZooKeeper. > > Since you are using Kafka version 3.3.1, you can leverage Kafka's built-in > consumer lag monitoring functionality without the need for any additional > tools or dependencies. Kafka exposes consumer lag metrics through JMX, and > you can use any monitoring system that supports JMX to collect and > visualize these metrics. > > Here are the steps to monitor consumer lag in Kafka: > > 1. > > Enable JMX on Kafka brokers and consumers by setting the following JVM > system properties: > > -Dcom.sun.management.jmxremote=true > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.local.only=false > -Djava.rmi.server.hostname=<hostname> > > > 1. > > Replace <hostname> with the hostname or IP address of the Kafka broker > or consumer machine. > 2. > > Start a monitoring system that supports JMX, such as Prometheus, > Grafana, or Datadog. > 3. > > Configure the monitoring system to collect Kafka consumer lag metrics > using JMX. > > For example, if you are using Prometheus, you can use the jmx_exporter > tool to expose the Kafka JMX metrics as Prometheus metrics. > 4. > > Visualize the Kafka consumer lag metrics in the monitoring system > dashboard. > > You can create custom dashboards that show the consumer lag for > different topics, consumers, and consumer groups. You can also set up > alerts to notify you when the consumer lag exceeds a certain threshold. > > By using Kafka's built-in JMX metrics and a monitoring system that supports > JMX, you can monitor consumer lag without the need for additional tools or > dependencies. This approach also provides a more lightweight and scalable > solution compared to using external tools like Burrow. > Thanks > Santhosh Gopal > Advisor - Architecture,Performance Engineering > > On Tue, May 9, 2023 at 2:14 PM Akshay Kumar <akshayku...@ameyo.com.invalid > > > wrote: > > > Hello team, > > > > I am using Zookeeper less Kafka (Kafka Kraft - version 3.3.1). I wanted > to > > monitor consumer lag, so I was using Burrow for that, but I am unable to > > use Burrow without Zookeeper. > > > > Does Burror work without Zookeeper? > > > > Or what is the better or best way to monitor consumer lag and lag history > > for multiple topics, consumers, and consumer groups? > > > > All the topics are consumed by different-different > consumers(applications). > > > > [image: https://exotel.com/] > > Akshay Kumar > > Member of Technical Staff - 2 | Exotel > > < > https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature > > > > [image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter] > > <https://twitter.com/Exotel> [image: Linkedin] > > <https://www.linkedin.com/company/exotel-techcom-private-limited/> > > Ph: +918556063696 > > Latest from the Exotel Blog > > < > https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/ > > > > > > [image: https://exotel.com/] <https://exotel.com/> > > *CONFIDENTIALITY NOTE:* This e-mail is intended only for the person or > > entity to which it is addressed and contains information that is > > privileged, confidential, or otherwise protected from disclosure. > > Dissemination, distribution, or copying of this e-mail or the information > > contained herein by anyone other than the intended recipient, or an > > employee or agent responsible for delivering the message to the intended > > recipient, is prohibited. If you have received this e-mail in error, > please > > delete this message and immediately notify the sender by e-mail. > > *NOTE:* This e-mail does not constitute an electronic signature and the > > sender does not intend to enter into any agreement by way of this e-mail, > > unless otherwise expressly provided by the sender within this e-mail. > > > -- <https://exotel.com/> *CONFIDENTIALITY NOTE:* This e-mail is intended only for the person or entity to which it is addressed and contains information that is privileged, confidential, or otherwise protected from disclosure. Dissemination, distribution, or copying of this e-mail or the information contained herein by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is prohibited. If you have received this e-mail in error, please delete this message and immediately notify the sender by e-mail. *NOTE:* This e-mail does not constitute an electronic signature and the sender does not intend to enter into any agreement by way of this e-mail, unless otherwise expressly provided by the sender within this e-mail.
