Re: Metrics for Topic/Partition size

2020-02-26 Thread Richard Rossel
awesome, thanks Gabriele On Wed, Feb 26, 2020 at 1:24 PM Gabriele Paggi wrote: > > Hi Richard, > > Yes, it's the size in bytes for all log segments for a given > topic/partition on a given broker, without the index files: > > [gpaggi@kafkalog001 ~]$ kafka-log-dirs.sh --bootstrap-server > $(hostna

Re: Metrics for Topic/Partition size

2020-02-26 Thread Gabriele Paggi
Hi Richard, Yes, it's the size in bytes for all log segments for a given topic/partition on a given broker, without the index files: [gpaggi@kafkalog001 ~]$ kafka-log-dirs.sh --bootstrap-server $(hostname -f):9092 --describe --broker-list 1 --topic-list access_logs | tail -n+3 | jq '.brokers[].l

Re: Metrics for Topic/Partition size

2020-02-26 Thread Richard Rossel
Thanks Gabriele, it turned out I didn't have that pattern deployed (facepalm). After deploying it , worked right away. Now I'm struggling with understanding the size metric. Do you know if it's reporting the size (in bytes) of all segments for that broker/topic/partition? I'm trying to compare tho

Re: Metrics for Topic/Partition size

2020-02-26 Thread Gabriele Paggi
Hi Richard, The beans path is: kafka.log:name=Size,partition=,topic=,type=Log I don't have a jmx_exporter at hand to test it at the moment but I don't see anything obviously wrong in your config, other than type: GAUGE missing. Did you try browsing the beans with jmxterm before configuring the exp

Re: Metrics for Topic/Partition size

2020-02-24 Thread Richard Rossel
Hi Gabriele, I'm using Kafka 5.3.1, which is apache kafka 2.3 and I'm using JMX to retrieve metrics from brokers. The only metric I saw from "kafka.log" is LogFlushStats, but nothing about Log.partition This is the pattern I have deployed for kafka.log metrics, maybe I need a different for partiti

Re: Metrics for Topic/Partition size

2020-02-24 Thread Gabriele Paggi
Hi Richard, If you are running Kafka > 1.0.0 the information you are looking for is exposed by the Admin API, describeLogDirs method, or via CLI with kafka-log-dirs.sh. The metrics are also exposed by the dropwizard metrics reporter, e.g. for Graphite: kafkalog001.kafka.log.Log.partition.49.topic.