Nice work. That tool I put together was getting a bit old. :-)
I updated the Kafka "ecosystem" page with details of both tools. https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem Philip ----------------------------------------- http://www.philipotoole.com On Wednesday, August 20, 2014 9:32 AM, Cory Watson <gp...@keen.io> wrote: Hello all! If you use Storm and Kafka 0.8 together I may have an interesting project for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7 to 0.8 and needed to replace the features of stormkafkamon <https://github.com/otoolep/stormkafkamon> for monitoring spout offsets since it only worked with Kafka 0.7. So we created Capillary <https://github.com/keenlabs/capillary>! Given a bit of information <https://github.com/keenlabs/capillary#configuration> about your Zookeeper setup for Storm and Kafka, Capillary is a web application that automatically discovers topologies running Kafka 0.8 spouts and lets you click on them to see per-partition offsets <https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>. It's JVM based and shouldn't require anything but Java where you run it. We decided this was a useful difference over the install-on-each-host-with-virtualenv model used by stormkafkamon. In addition to the helpful UI there is an API that you might use for collecting this information and sending it to your stats/observability systems for historical or alerting purposes. There's even an example python script <https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we use for reporting to Datadog <http://www.datadoghq.com/>. It's very likely we'll move this to a scheduled thread inside Capillary using metrics <http://metrics.codahale.com/> and allow you to hook up your own reporter. I hope other uses of Kafka + Storm find this useful. Feel free to open an issue if you have issues setting it up or configuring it, or drop PRs if you have additional functionality. Thanks! (I'll be cross-posting this to the Storm list also!) -- Cory Watson Principal Infrastructure Engineer // Keen IO
