Re: Anouncing Kafka Offset Monitor 0.1

2014-03-10 Thread Otis Gospodnetic
Interesting. We were able to get this from JMX for Kafka 0.7.2 - here's a snapshot for one of our Kafka clusters: https://apps.sematext.com/spm-reports/s/93EtvhnOz0 Is getting this from ZK instead of JMX better? Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsear

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-08 Thread Pierre Andrews
It's been developed with 0.8 in mind. Giving a quick look at how 0.7 stores data in ZK, it looks like it's slightly different and might not work straight away. If you are intersted, have a look at: https://github.com/quantifind/KafkaOffsetMonitor/blob/master/src/main/scala/com/quantifind/kafka/Off

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-08 Thread Pierre Andrews
Hi Patricio, thank you! I don't know how kafka-spout works. Right now, the monitor uses the value defined in kafka.utils.ZkUtils.ConsumersPath to grab the list of consumers in ZK. Again, that could be a parameter to the app somewhere to be more flexible and deal with separate zk path. P On Sat

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-08 Thread Pierre Andrews
On Sat, Mar 8, 2014 at 12:17 AM, Dan Hoffman wrote: > Hi - am I right in that for this tool to be effective, consumers must be > using the high level consumer or otherwise keeping their offsets in > zookeeper? yes indeed, this will look into zookeeper for the consumers and their offsets. > I

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Otis Gospodnetic
Hi, Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info needed for computing the lag? Thanks, Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews wrote: > Hello

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Azuryy Yu
hi, Kafka web console will be included in the next release. On Mar 8, 2014 2:49 AM, "Pierre Andrews" wrote: > Hello everyone, > > at Quantifind, we are big users of Kafka and we like it a lot! > In a few use cases, we had to figure out if a queue was growing and how its > consumers were behavin

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Patricio Echagüe
Great work! In addition to Dan's question, does it work with storm kafka-spout which uses a separate zk path? Sent from my Nexus 4. On Mar 7, 2014 7:18 PM, "Dan Hoffman" wrote: > Hi - am I right in that for this tool to be effective, consumers must be > using the high level consumer or otherwis

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Dan Hoffman
Hi - am I right in that for this tool to be effective, consumers must be using the high level consumer or otherwise keeping their offsets in zookeeper? Is there any way to track performance without that? On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin wrote: > Very nice > > > On Mar 7, 2014, at 11

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Steve Morin
Very nice > On Mar 7, 2014, at 11:55, Pierre Andrews wrote: > > Claude, we should join forces ;) > > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo wrote: >> >> Awesome!!! ;-) >> >> Claude >> >> >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews >> wrote: >> >>> Great! Thanks! >>> >>> >

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Claude Mamo
Agreed, sent you an email. Caude On Fri, Mar 7, 2014 at 4:55 PM, Pierre Andrews wrote: > Claude, we should join forces ;) > > > On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo wrote: > > > Awesome!!! ;-) > > > > Claude > > > > > > On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews > >wrote: > > > > >

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Pierre Andrews
Claude, we should join forces ;) On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo wrote: > Awesome!!! ;-) > > Claude > > > On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews >wrote: > > > Great! Thanks! > > > > > > On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps wrote: > > > > > This is really useful! I add

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Claude Mamo
Awesome!!! ;-) Claude On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews wrote: > Great! Thanks! > > > On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps wrote: > > > This is really useful! I added it to the ecosystem page: > > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem > > > > -Jay > > >

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Pierre Andrews
Great! Thanks! On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps wrote: > This is really useful! I added it to the ecosystem page: > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem > > -Jay > > > On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews >wrote: > > > Hello everyone, > > > > at Quant

Re: Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Jay Kreps
This is really useful! I added it to the ecosystem page: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem -Jay On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews wrote: > Hello everyone, > > at Quantifind, we are big users of Kafka and we like it a lot! > In a few use cases, we had to f

Anouncing Kafka Offset Monitor 0.1

2014-03-07 Thread Pierre Andrews
Hello everyone, at Quantifind, we are big users of Kafka and we like it a lot! In a few use cases, we had to figure out if a queue was growing and how its consumers were behaving. There are a few command-line tools to try to figure out what's going on, but it's not always easy to debug and to see