Are you using Kafka based offset commit or ZK based offset commit? On 2/28/15, 6:16 AM, "Gene Robichaux" <gene.robich...@match.com> wrote:
>What is the best way to detect consumer lag? > >We are running each consumer as a separate group and I am running the >ConsumerOffsetChecker to assess the partitions and the lag for each >group/consumer. I run this every 5 minutes. In some cases I run this >command up to 75 times on each 5 min polling cycle (once for each >group/consuer). An example of the command is (bin/kafka-run-class.sh >kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect >zkhost:zkport) > >The problem I am running into is CPU usage on the broker when these >commands run. We have a dedicated broker that has no leader partitions, >but the high CPU still concerns me. > >Is there a better way to detect consumer lag? Preferably one that is less >impactful? > > >Gene Robichaux >Manager, Database Operations >Match.com >8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 >
