Thank you Brett I identified this metrics as well, thank you for confirming. I ended up using TotalProduceRequestsPerSec and TotalFetchRequestsPerSec. Regards
On Tue, Aug 21, 2018 at 9:53 PM Brett Rann <br...@zendesk.com.invalid> wrote: > These are in regex form for DataDog's JMX collector, but it should get you > started: > > bean_regex: > > 'kafka\.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=.*' > bean_regex: > > 'kafka\.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=.*' > bean_regex: > 'kafka\.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=.*' > bean_regex: > 'kafka\.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=.*' > bean_regex: > 'kafka\.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=.*' > bean_regex: > > 'kafka\.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic=.*' > bean_regex: > > 'kafka\.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic=.*' > bean_regex: > > 'kafka\.server:type=BrokerTopicMetrics,name=ProduceMessageConversionsPerSec,topic=.*' > bean_regex: > > 'kafka\.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=.*' > > On Wed, Aug 22, 2018 at 9:24 AM Luigi Tagliamonte < > luigi.tagliamont...@gmail.com> wrote: > > > I've got a sandbox cluster (without any control on topic creation) full > of > > unused topics and I'd like to cleanup all the those topics. > > I was able to identify all the topics with 0 messages with the following > > lines: > > > > *#!/bin/bash* > > *set -e* > > *BROKERS="10.16.208.220:9092 <http://10.16.208.220:9092 > > <http://10.16.208.220:9092>>, > > 10.16.209.24:9092 > > <http://10.16.209.24:9092 > > <http://10.16.209.24:9092>>, > > 10.16.209.146:9092 <http://10.16.209.146:9092 > > <http://10.16.209.146:9092>>"* > > *ZOOKEEPER="10.16.209.44:2181 <http://10.16.209.44:2181 > > <http://10.16.209.44:2181>>"* > > *declare -a topics=($(/opt/kafka/bin/kafka-topics.sh --zookeeper > > ${ZOOKEEPER} --list | tr "\n" " "))* > > *for topic in ${topics[@]}* > > *do* > > * messages=$(/opt/kafka/bin/kafka-run-class.sh > > kafka.tools.GetOffsetShell --broker-list ${BROKERS} --topic $topic --time > > -1| tr ":" " " | awk '{ sum += $3 } END { print sum }')* > > * echo "$topic has $messages messages"* > > *done* > > > > but I'd like to sample also the topic usage, and do a match between the 2 > > lists. > > Is there any JMX metric I can use to make sure nobody is > receiving/creating > > messages for each topic like a total counter or read/write per topic? > > Cassandra has something like that so I was wondering if Kafka has > something > > like that as well. > > Looking forward to your reply. > > Regards > > L. > > >
