I think we ZK based offset commit. However I am not certain, I would have to get that from our DEV group. My role is PROD Ops.
Gene Robichaux Manager, Database Operations Match.com 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 -----Original Message----- From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID] Sent: Saturday, February 28, 2015 12:06 PM To: users@kafka.apache.org Subject: Re: Best way to show lag? Are you using Kafka based offset commit or ZK based offset commit? On 2/28/15, 6:16 AM, "Gene Robichaux" <gene.robich...@match.com> wrote: >What is the best way to detect consumer lag? > >We are running each consumer as a separate group and I am running the >ConsumerOffsetChecker to assess the partitions and the lag for each >group/consumer. I run this every 5 minutes. In some cases I run this >command up to 75 times on each 5 min polling cycle (once for each >group/consuer). An example of the command is (bin/kafka-run-class.sh >kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect >zkhost:zkport) > >The problem I am running into is CPU usage on the broker when these >commands run. We have a dedicated broker that has no leader partitions, >but the high CPU still concerns me. > >Is there a better way to detect consumer lag? Preferably one that is >less impactful? > > >Gene Robichaux >Manager, Database Operations >Match.com >8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 >
