Hello everyone, at Quantifind, we are big users of Kafka and we like it a lot! In a few use cases, we had to figure out if a queue was growing and how its consumers were behaving. There are a few command-line tools to try to figure out what's going on, but it's not always easy to debug and to see what has happened while everyone was sleeping.
To be able to monitor our kafka queues and consumers, we thus developed a tiny web app that could tell us the log size of each topic in the brokers and the offsets of each consumers. That's very similar to what the kafka ConsumerOffsetChecker tool* is doing, but instead of having a one off snapshot, our app keeps an history and displays a nice graph of what's going on. You can find screenshots and more details here: http://quantifind.github.io/KafkaOffsetMonitor/ the code is on github: https://github.com/quantifind/KafkaOffsetMonitor If you have kafka 0.8 setup, it's very easy to use: 1- download the current jar http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar 2- run it, pointing at your kafka brokers: java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \ com.quantifind.kafka.offsetapp.OffsetGetterWeb \ --zk zk-server1,zk-server2 \ --port 8080 \ --refresh 10.seconds \ --retain 2.days 3- open your browser and point it to localhost:8080 You can run it locally or host it on a server if you prefer. It's all open source and we'll be happy to receive issue report and pull requests. I hope that you like it and that it can find some uses in the rest of the Kafka community. Best Pierre PS: we are aware of https://github.com/claudemamo/kafka-web-console but are currently offering slightly different features. Hopefully we can merge the projects in the future. * here: https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
