If you are using ZK based offset commit, you have to read the offset from ZK. If you can make code change, one potential improvement is to reuse ZKClient as explained below. Currently, ConsumerOffsetChecker only takes one consumer group for each run. If you have a lot of consumer groups to check, every time you run the tool, it will create another ZKClient, which is not very efficient in your use case. If your dev can make some minor code change to let the tool take a list of consumer group and reuse ZKClient instead of creating it in each run. Can you verify if this solution works? If it works, can you also create an JIRA ticket and submit the patch? I think this might also be a good improvement for Kafka based offset commit.
Jiangjie (Becket) Qin On 3/1/15, 5:44 AM, "Gene Robichaux" <gene.robich...@match.com> wrote: >That is what I am using. The problem is when I run it the CPU spikes on >the broker I am running it from. I just wanted to know if there was a >different way. > >Gene > >Sent from my iPhone > >> On Feb 28, 2015, at 10:46 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> >> If it is ZK based offset commit, you can use the ConsumerOffsetChecker >>tool >> in kafka.tools. >> >> On Sat, Feb 28, 2015 at 12:32 PM, Gene Robichaux >><gene.robich...@match.com> >> wrote: >> >>> I think we ZK based offset commit. However I am not certain, I would >>>have >>> to get that from our DEV group. My role is PROD Ops. >>> >>> Gene Robichaux >>> Manager, Database Operations >>> Match.com >>> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 >>> >>> -----Original Message----- >>> From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID] >>> Sent: Saturday, February 28, 2015 12:06 PM >>> To: users@kafka.apache.org >>> Subject: Re: Best way to show lag? >>> >>> Are you using Kafka based offset commit or ZK based offset commit? >>> >>>> On 2/28/15, 6:16 AM, "Gene Robichaux" <gene.robich...@match.com> >>>>wrote: >>>> >>>> What is the best way to detect consumer lag? >>>> >>>> We are running each consumer as a separate group and I am running the >>>> ConsumerOffsetChecker to assess the partitions and the lag for each >>>> group/consumer. I run this every 5 minutes. In some cases I run this >>>> command up to 75 times on each 5 min polling cycle (once for each >>>> group/consuer). An example of the command is (bin/kafka-run-class.sh >>>> kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect >>>> zkhost:zkport) >>>> >>>> The problem I am running into is CPU usage on the broker when these >>>> commands run. We have a dedicated broker that has no leader >>>>partitions, >>>> but the high CPU still concerns me. >>>> >>>> Is there a better way to detect consumer lag? Preferably one that is >>>> less impactful? >>>> >>>> >>>> Gene Robichaux >>>> Manager, Database Operations >>>> Match.com >>>> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 >> >> >> -- >> -- Guozhang
