Chen, Your configs seems fine.
Could you use ConsumerOffsetChecker tool to see if the offset is advancing at all (i.e. messages are comsumed), and if yes get some thread dumps and check if your consumer is blocked on some locks? Guozhang On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang <chen.apache.s...@gmail.com> wrote: > Hey Guys, > I have a really simply storm topology with a kafka spout, reading from > kafka through high level consumer. Since the topic has 30 partitions, we > have 30 threads in the spout reading from it. However, it seems that the > lag keeps increasing even the thread only read the message and do nothing. > The largest message size are around 30KB, and the incoming rate can be as > hight as 14k/seconds. There are 3 brokers on some high config bare metal > machines. The client side config is like this: > > kafka.config.fetch.message.max.bytes 3145728 > kafka.config.group.id spout_readonly > kafka.config.rebalance.backoff.ms 6000 > kafka.config.rebalance.max.retries 6 > kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181, > dare-broker01.sv.walmartlabs.com:2181, > dare-broker02.sv.walmartlabs.com:2181 > kafka.config.zookeeper.session.timeout.ms 60000 > > what could possibly cause this huge lag? Will broker be a bottle neck, or > some config need to be adjusted? The server side config is like this: > > replica.fetch.max.bytes=2097152 > message.max.bytes=2097152 > num.network.threads=4 > num.io.threads=4 > > # The send buffer (SO_SNDBUF) used by the socket server > socket.send.buffer.bytes=4194304 > > # The receive buffer (SO_RCVBUF) used by the socket server > socket.receive.buffer.bytes=2097152 > > # The maximum size of a request that the socket server will accept > (protection against OOM) > socket.request.max.bytes=104857600 > > Any help appreciated! > Chen > -- -- Guozhang
