Chen,
Consumers lag either due to an I/O or network bottleneck or due to slow
processing of messages by the user. To confirm that you are not hitting the
latter issue, you can run a console consumer on the same data and observe
the throughput that it provides and it's lag.
Thanks,
Neha
On Wed, N
Guozhang,
I can see message keep coming, meaning messages are being consumed, right?
But the lag is pretty huge (average 30m messages behind) as you can see
from the graph:
https://www.dropbox.com/s/xli25zicxv5f2qa/Screenshot%202014-11-05%2015.23.05.png?dl=0
My understanding is that for such light
Chen,
Your configs seems fine.
Could you use ConsumerOffsetChecker tool to see if the offset is advancing
at all (i.e. messages are comsumed), and if yes get some thread dumps and
check if your consumer is blocked on some locks?
Guozhang
On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang
wrote:
> Hey
Hey Guys,
I have a really simply storm topology with a kafka spout, reading from
kafka through high level consumer. Since the topic has 30 partitions, we
have 30 threads in the spout reading from it. However, it seems that the
lag keeps increasing even the thread only read the message and do nothin
