These are in regex form for DataDog's JMX collector, but it should get you
started:
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=ProduceMessageConversionsPerSec,topic=.*'
bean_regex:
'kafka\.server:type=BrokerTopicMetrics,name=FetchMessageConversionsPerSec,topic=.*'
On Wed, Aug 22, 2018 at 9:24 AM Luigi Tagliamonte <
luigi.tagliamont...@gmail.com> wrote:
> I've got a sandbox cluster (without any control on topic creation) full of
> unused topics and I'd like to cleanup all the those topics.
> I was able to identify all the topics with 0 messages with the following
> lines:
>
> *#!/bin/bash*
> *set -e*
> *BROKERS="10.16.208.220:9092 <http://10.16.208.220:9092
> <http://10.16.208.220:9092>>,
> 10.16.209.24:9092
> <http://10.16.209.24:9092
> <http://10.16.209.24:9092>>,
> 10.16.209.146:9092 <http://10.16.209.146:9092
> <http://10.16.209.146:9092>>"*
> *ZOOKEEPER="10.16.209.44:2181 <http://10.16.209.44:2181
> <http://10.16.209.44:2181>>"*
> *declare -a topics=($(/opt/kafka/bin/kafka-topics.sh --zookeeper
> ${ZOOKEEPER} --list | tr "\n" " "))*
> *for topic in ${topics[@]}*
> *do*
> * messages=$(/opt/kafka/bin/kafka-run-class.sh
> kafka.tools.GetOffsetShell --broker-list ${BROKERS} --topic $topic --time
> -1| tr ":" " " | awk '{ sum += $3 } END { print sum }')*
> * echo "$topic has $messages messages"*
> *done*
>
> but I'd like to sample also the topic usage, and do a match between the 2
> lists.
> Is there any JMX metric I can use to make sure nobody is receiving/creating
> messages for each topic like a total counter or read/write per topic?
> Cassandra has something like that so I was wondering if Kafka has something
> like that as well.
> Looking forward to your reply.
> Regards
> L.
>
