Re: Consumer Lag increasing but offset not moving.

2013-08-14 Thread Ramakrishna Mallya
Yes that is the log for one of the consumers, but it is same for the other consumers too that we have running for this queue. We have 4 consumers running with 300 streams each to match the partitions on the brokers. And yes the backlog is growing and offsets not moving. Following is the sampling t

Re: Consumer Lag increasing but offset not moving.

2013-08-14 Thread Joel Koshy
So those are the logs for one consumer right? How many consumer instances do you have? Can you run the offset checker tool at an interval greater than the offset commit interval and verify that offsets are not moving and that log sizes are growing? https://cwiki.apache.org/confluence/display/KAFKA

Consumer Lag increasing but offset not moving.

2013-08-14 Thread Ramakrishna Mallya
Hi, We are using kafka-0.7.2 with zookeeper (3.4.5) Our cluster configuration: 3 brokers on 3 different machines. Each broker machine has a zookeeper instance running as well. We are currently facing a problem with messages not being consumed by the kafka high level consumers. This is a dedicate