What do you mean by "We have a dedicated broker that has no leader partitions". Are you running anything else on that machine? I think you can run that tool Guozhang from any machine and don't require it to be a kafka Broker.
Thanks, Mayuresh On Sun, Mar 1, 2015 at 5:44 AM, Gene Robichaux <gene.robich...@match.com> wrote: > That is what I am using. The problem is when I run it the CPU spikes on > the broker I am running it from. I just wanted to know if there was a > different way. > > Gene > > Sent from my iPhone > > > On Feb 28, 2015, at 10:46 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > > If it is ZK based offset commit, you can use the ConsumerOffsetChecker > tool > > in kafka.tools. > > > > On Sat, Feb 28, 2015 at 12:32 PM, Gene Robichaux < > gene.robich...@match.com> > > wrote: > > > >> I think we ZK based offset commit. However I am not certain, I would > have > >> to get that from our DEV group. My role is PROD Ops. > >> > >> Gene Robichaux > >> Manager, Database Operations > >> Match.com > >> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 > >> > >> -----Original Message----- > >> From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID] > >> Sent: Saturday, February 28, 2015 12:06 PM > >> To: users@kafka.apache.org > >> Subject: Re: Best way to show lag? > >> > >> Are you using Kafka based offset commit or ZK based offset commit? > >> > >>> On 2/28/15, 6:16 AM, "Gene Robichaux" <gene.robich...@match.com> > wrote: > >>> > >>> What is the best way to detect consumer lag? > >>> > >>> We are running each consumer as a separate group and I am running the > >>> ConsumerOffsetChecker to assess the partitions and the lag for each > >>> group/consumer. I run this every 5 minutes. In some cases I run this > >>> command up to 75 times on each 5 min polling cycle (once for each > >>> group/consuer). An example of the command is (bin/kafka-run-class.sh > >>> kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect > >>> zkhost:zkport) > >>> > >>> The problem I am running into is CPU usage on the broker when these > >>> commands run. We have a dedicated broker that has no leader partitions, > >>> but the high CPU still concerns me. > >>> > >>> Is there a better way to detect consumer lag? Preferably one that is > >>> less impactful? > >>> > >>> > >>> Gene Robichaux > >>> Manager, Database Operations > >>> Match.com > >>> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 > > > > > > -- > > -- Guozhang > -- -Regards, Mayuresh R. Gharat (862) 250-7125
