Whoops, just spotted a typo - the second $1 in the above snippet should of course be $2.
On Thu, Jul 16, 2020 at 4:33 PM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Pushkar, > > There are broker side metrics for messages in / bytes in / bytes out per > topic per second. I use this jmx_exporter rule to export them: > - pattern: 'kafka.server<type=BrokerTopicMetrics, name=(.+), > topic=(.+)><>OneMinuteRate' > name: kafka_broker_per_topic_$1_one_minute_rate > labels: > topic: $1 > type: GAUGE > > You can't get the number of messages out per topic from the broker > because... I think it's somehow related to batching, or similar, it doesn't > count messages out, only bytes out. You can, however, get that metric from > the consumer if you're using the Java Kafka client, there's a per-topic > messages consumed per second metric exposed as an MBean. > > You could either use jmx_exporter to also export that from the client app, > or if possible, add some code that connects to the mbean inside the JVM and > then exports it via any pre-existing Prometheus registry. You might want to > then use a Prometheus aggregating rule to collate all the per-consumer apps > into a per-consumer-group metric, unless the per consumer granularity is of > interest to you: https://prometheus.io/docs/practices/rules/ > > Hope that helps, > > Kind regards, > > Liam Clarke-Hutchinsons > > On Thu, 16 Jul. 2020, 3:46 pm Pushkar Deole, <pdeole2...@gmail.com> wrote: > >> Thanks Claudia! For broker level metrics, we are also using same jmx >> exporter to export those metrics to prometheus. >> Are you fetching any per topic metrics from broker? e.g. messages produced >> on a certain topic or messages consumed from a certain topic. I am mainly >> interested in these metrics. >> >> I read in kafka docs that they are present at producer/consumer, however I >> am not sure how to fetch them from consumer/producer. >> >> On Wed, Jul 15, 2020 at 8:32 PM Claudia Kesslau <c.kess...@kasasi.de> >> wrote: >> >> > Hi, >> > >> > I use https://github.com/prometheus/jmx_exporter for collecting broker >> > metrics and integrating them into prometheus. >> > >> > Hope this helps. >> > Greetings, >> > Claudia >> > >> > -----Ursprüngliche Nachricht----- >> > Von: Pushkar Deole <pdeole2...@gmail.com> >> > Gesendet: Mittwoch, 15. Juli 2020 09:07 >> > An: users@kafka.apache.org >> > Betreff: Re: kafka per topic metrics >> > >> > We are using prometheus as metrics collection and storage system and >> > Grafana for displaying those metrics, so integration with them is >> required >> > >> > On Wed, Jul 15, 2020 at 11:11 AM rohit garg <rohit.gar...@gmail.com> >> > wrote: >> > >> > > You can try using kafka manager and check it will fullfill most of >> > > requirement of yours. >> > > >> > > Thanks and Regards, >> > > Rohit >> > > >> > > On Wed, Jul 15, 2020, 10:33 Pushkar Deole <pdeole2...@gmail.com> >> wrote: >> > > >> > > > Hi All, >> > > > >> > > > Any inputs as to how the kafka consumer and producer metrics can be >> > > hooked >> > > > up to a monitoring system such as prometheus ? >> > > > >> > > > On Tue, Jul 14, 2020 at 4:22 PM Pushkar Deole <pdeole2...@gmail.com >> > >> > > > wrote: >> > > > >> > > > > i did find these metrics from confluent docs: however how can i >> > > > > get >> > > this >> > > > > metric ? is it available at kafka broker? >> > > > > >> > > > > Per-Topic Metrics >> > > > > >> > > > > MBean: >> > > > > >> > > > >> > > kafka.producer:type=producer-topic-metrics,client-id=([-.w]+),topic=([ >> > > -.w]+) >> > > > > record-send-rateThe average number of records sent per second for >> > > > > a >> > > > topic. >> > > > > >> > > > > On Tue, Jul 14, 2020 at 3:27 PM Pushkar Deole >> > > > > <pdeole2...@gmail.com> >> > > > > wrote: >> > > > > >> > > > >> Hi All, >> > > > >> >> > > > >> Need some help on kafka metrics, i am interested in certain >> > > > >> metrics >> > > e.g. >> > > > >> i need to know the number of records published on a particular >> > > > >> topic >> > > and >> > > > >> number of records consumed from that topic by a specific consumer >> > > > group, i >> > > > >> would need a total of these 2 and also average per second for >> them. >> > > > >> >> > > > >> Are those metrics available on kafka brokers and if yes then >> > > > >> which are those metrics that would give me above counts? >> > > > >> >> > > > > >> > > > >> > > >> > >> >
