Guozhang, I can see message keep coming, meaning messages are being consumed, right? But the lag is pretty huge (average 30m messages behind) as you can see from the graph: https://www.dropbox.com/s/xli25zicxv5f2qa/Screenshot%202014-11-05%2015.23.05.png?dl=0
My understanding is that for such light weight thread, the consumer should almost be at the same pace with the producer. I also checked the machine metrics, and nothing pegged there. I am also moving the testing application to a separate dev cluster. In your experience, what things might cause the slow reading? Is this more like a server side thing, or consumer side? Chen On Wed, Nov 5, 2014 at 3:10 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Chen, > > Your configs seems fine. > > Could you use ConsumerOffsetChecker tool to see if the offset is advancing > at all (i.e. messages are comsumed), and if yes get some thread dumps and > check if your consumer is blocked on some locks? > > Guozhang > > On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang <chen.apache.s...@gmail.com> > wrote: > > > Hey Guys, > > I have a really simply storm topology with a kafka spout, reading from > > kafka through high level consumer. Since the topic has 30 partitions, we > > have 30 threads in the spout reading from it. However, it seems that the > > lag keeps increasing even the thread only read the message and do > nothing. > > The largest message size are around 30KB, and the incoming rate can be > as > > hight as 14k/seconds. There are 3 brokers on some high config bare metal > > machines. The client side config is like this: > > > > kafka.config.fetch.message.max.bytes 3145728 > > kafka.config.group.id spout_readonly > > kafka.config.rebalance.backoff.ms 6000 > > kafka.config.rebalance.max.retries 6 > > kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181, > > dare-broker01.sv.walmartlabs.com:2181, > > dare-broker02.sv.walmartlabs.com:2181 > > kafka.config.zookeeper.session.timeout.ms 60000 > > > > what could possibly cause this huge lag? Will broker be a bottle neck, or > > some config need to be adjusted? The server side config is like this: > > > > replica.fetch.max.bytes=2097152 > > message.max.bytes=2097152 > > num.network.threads=4 > > num.io.threads=4 > > > > # The send buffer (SO_SNDBUF) used by the socket server > > socket.send.buffer.bytes=4194304 > > > > # The receive buffer (SO_RCVBUF) used by the socket server > > socket.receive.buffer.bytes=2097152 > > > > # The maximum size of a request that the socket server will accept > > (protection against OOM) > > socket.request.max.bytes=104857600 > > > > Any help appreciated! > > Chen > > > > > > -- > -- Guozhang >
