Hey Guys, I have a really simply storm topology with a kafka spout, reading from kafka through high level consumer. Since the topic has 30 partitions, we have 30 threads in the spout reading from it. However, it seems that the lag keeps increasing even the thread only read the message and do nothing. The largest message size are around 30KB, and the incoming rate can be as hight as 14k/seconds. There are 3 brokers on some high config bare metal machines. The client side config is like this:
kafka.config.fetch.message.max.bytes 3145728 kafka.config.group.id spout_readonly kafka.config.rebalance.backoff.ms 6000 kafka.config.rebalance.max.retries 6 kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181, dare-broker01.sv.walmartlabs.com:2181,dare-broker02.sv.walmartlabs.com:2181 kafka.config.zookeeper.session.timeout.ms 60000 what could possibly cause this huge lag? Will broker be a bottle neck, or some config need to be adjusted? The server side config is like this: replica.fetch.max.bytes=2097152 message.max.bytes=2097152 num.network.threads=4 num.io.threads=4 # The send buffer (SO_SNDBUF) used by the socket server socket.send.buffer.bytes=4194304 # The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=2097152 # The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600 Any help appreciated! Chen
