Hi - am I right in that for this tool to be effective, consumers must be using the high level consumer or otherwise keeping their offsets in zookeeper? Is there any way to track performance without that?
On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <steve.mo...@gmail.com> wrote: > Very nice > > > On Mar 7, 2014, at 11:55, Pierre Andrews <pie...@quantifind.com> wrote: > > > > Claude, we should join forces ;) > > > > > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <claude.m...@gmail.com> > wrote: > >> > >> Awesome!!! ;-) > >> > >> Claude > >> > >> > >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pie...@quantifind.com > >>> wrote: > >> > >>> Great! Thanks! > >>> > >>> > >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <jay.kr...@gmail.com> > wrote: > >>>> > >>>> This is really useful! I added it to the ecosystem page: > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem > >>>> > >>>> -Jay > >>>> > >>>> > >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews < > pie...@quantifind.com > >>>>> wrote: > >>>> > >>>>> Hello everyone, > >>>>> > >>>>> at Quantifind, we are big users of Kafka and we like it a lot! > >>>>> In a few use cases, we had to figure out if a queue was growing and > >> how > >>>> its > >>>>> consumers were behaving. There are a few command-line tools to try to > >>>>> figure out what's going on, but it's not always easy to debug and to > >>> see > >>>>> what has happened while everyone was sleeping. > >>>>> > >>>>> To be able to monitor our kafka queues and consumers, we thus > >>> developed a > >>>>> tiny web app that could tell us the log size of each topic in the > >>> brokers > >>>>> and the offsets of each consumers. That's very similar to what the > >>> kafka > >>>>> ConsumerOffsetChecker tool* is doing, but instead of having a one off > >>>>> snapshot, our app keeps an history and displays a nice graph of > >> what's > >>>>> going on. > >>>>> > >>>>> You can find screenshots and more details here: > >>>>> http://quantifind.github.io/KafkaOffsetMonitor/ > >>>>> > >>>>> the code is on github: > >>>>> https://github.com/quantifind/KafkaOffsetMonitor > >>>>> > >>>>> If you have kafka 0.8 setup, it's very easy to use: > >>>>> > >>>>> 1- download the current jar > >> > http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar > >>>>> 2- run it, pointing at your kafka brokers: > >>>>> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \ > >>>>> com.quantifind.kafka.offsetapp.OffsetGetterWeb \ > >>>>> --zk zk-server1,zk-server2 \ > >>>>> --port 8080 \ > >>>>> --refresh 10.seconds \ > >>>>> --retain 2.days > >>>>> 3- open your browser and point it to localhost:8080 > >>>>> > >>>>> You can run it locally or host it on a server if you prefer. > >>>>> > >>>>> It's all open source and we'll be happy to receive issue report and > >>> pull > >>>>> requests. > >>>>> > >>>>> I hope that you like it and that it can find some uses in the rest of > >>> the > >>>>> Kafka community. > >>>>> > >>>>> Best > >>>>> > >>>>> Pierre > >>>>> > >>>>> PS: we are aware of > >> https://github.com/claudemamo/kafka-web-consolebut > >>>>> are > >>>>> currently offering slightly different features. Hopefully we can > >> merge > >>>> the > >>>>> projects in the future. > >>>>> > >>>>> * here: > >> > https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala > >> >
