I’m going to say no... Metric beat expose OS metrics. You talking about a JVM here exposing values via JMX.
Have you looked at catching all of this using Prometheus and dash boarding it using Grafana, as a real time dashboard. Sent from my iPhone George Leonard __________________________ george...@gmail.com +27 82 655 2466 > On 21 Feb 2020, at 07:16, Sunil CHAUDHARI <sunilchaudh...@dbs.com.invalid> > wrote: > > Hi Liam Clarke, > Thanks for this elaboration. > Surely I will google. > One more question about Mbeans. If I am capturing system matrix on kafka > broker using metricbeat, then is it possible that I will get those Mbeans? > I know this is out of the topic, but in general, if I capture JVM metric with > any of the monitoring tools, then will I get it? > > > > -----Original Message----- > From: Liam Clarke <liam.cla...@adscale.co.nz> > Sent: Friday, February 21, 2020 1:03 AM > To: users@kafka.apache.org > Subject: Re: [External] Re: Urgent helep please! How to measure producer and > consumer throughput for single partition? > > This mail originated from an external party outside DBS - > users-return-39570-sunilchaudhari=dbs....@kafka.apache.org. Do not click on > links or open attachments unless you recognize the sender and know the > content is safe. > > The metrics are exposed in the JVM the producer is running within as Mbeans. > The long string I gave you is the relevant MBean object name. You can connect > to the JVM using JConsole to view the MBeans. There are also multiple > libraries that will scrape a JVM via JMX to extract values from MBeans. > > If you're not familiar with JMX or JConsole, there's plenty of great > documentation on the Internet, have a Google :) > >> On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, >> <sunilchaudh...@dbs.com.invalid> wrote: >> >> Hi Liam Clarke, >> Sorry but this is bit unclear for me. >> Can you please elaborate your answer? I am beginner to Kafka. >> " Producers emit metrics via JMX ": >> - How to enable this? I have kafka-Manager. Can I make use of >> kafka-manager? How? >> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate” >> please help to explain this. >> >> Regards, >> Sunil. >> >> -----Original Message----- >> From: Liam Clarke <liam.cla...@adscale.co.nz> >> Sent: Thursday, February 20, 2020 11:16 AM >> To: users@kafka.apache.org >> Subject: [External] Re: Urgent helep please! How to measure producer >> and consumer throughput for single partition? >> >> This mail originated from an external party outside DBS - mailto: >> users-return-39554-sunilchaudhari=dbs....@kafka.apache.org. Do not >> click on links or open attachments unless you recognize the sender and >> know the content is safe. >> >> Hi Sunil, >> >> Producers emit metrics via JMX that will help you, assuming that your >> producers are using a round robin partition assignment strategy, you >> could divide this metric by your number of partitions, >> >> >> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s >> end-rate >> >> Kind regards, >> >> Liam Clarke >> >> On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto: >> sunilchaudh...@dbs.com.invalid> wrote: >> >>> Hi >>> I was referring to the article by Mr. June Rao about partitions in >>> kafka cluster. >>> https://www.confluent.io/blog/how-choose-number-topics-partitions-ka >>> fk >>> a-cluster/ >>> >>> "A rough formula for picking the number of partitions is based on >>> throughput. You measure the throughout that you can achieve on a >>> single partition for production (call it p) and consumption (call it >>> c). Let's say your target throughput is t. Then you need to have at >>> least max(t/p, t/c) partitions." >>> >>> I have the data pipeline as below. >>> >>> Filebeat-->Kafka-->Logstash-->Elasticsearch >>> There are many filebeat agents sending data to kafka. I want to >>> understand , how can I measure the events per seconds getting >>> written to Kafka? This will help me to know 'p' in above formula. >>> I can measure the consumer throughput by monitoring logsatsh >>> pipelines on Kibana. So it will give me 'c' in above formula. >>> >>> I know target throughput in my cluster, that is 't'. 30k events/s. >>> >>> Please let me know if I am going wrong? >>> >>> Regards, >>> Sunil. >>> CONFIDENTIAL NOTE: >>> The information contained in this email is intended only for the use >>> of the individual or entity named above and may contain information >>> that is privileged, confidential and exempt from disclosure under >> applicable law. >>> If the reader of this message is not the intended recipient, you are >>> hereby notified that any dissemination, distribution or copying of >>> this communication is strictly prohibited. If you have received this >>> message in error, please immediately notify the sender and delete >>> the >> mail. Thank you. >>> >> >> >> CONFIDENTIAL NOTE: >> The information contained in this email is intended only for the use >> of the individual or entity named above and may contain information >> that is privileged, confidential and exempt from disclosure under applicable >> law. >> If the reader of this message is not the intended recipient, you are >> hereby notified that any dissemination, distribution or copying of >> this communication is strictly prohibited. If you have received this >> message in error, please immediately notify the sender and delete the mail. >> Thank you. >> > > CONFIDENTIAL NOTE: > The information contained in this email is intended only for the use of the > individual or entity named above and may contain information that is > privileged, confidential and exempt from disclosure under applicable law. If > the reader of this message is not the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you have received this message in > error, please immediately notify the sender and delete the mail. Thank you.
