Kafka New(Java) Producer Connection reset by peer error and LB

[Bhavesh Mistry]

Hi Kakfa Team,

We are getting this connection reset by pears after couple of minute aster
start-up of producer due to infrastructure deployment strategies we have
adopted from LinkedIn.

We have LB hostname and port as seed server, and all producers are getting
following exception because of TCP idle connection timeout set on LB (which
is 2 minutes and Kafka TCP connection idle is set to 10 minutes).   This
seems to be  minor bug to close TCP connection after discovering that seed
server is not part of brokers list immediately.


java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
    at sun.nio.ch.IOUtil.read(IOUtil.java:171)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
    at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:60)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:242)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:178)
    at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
    at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
    at java.lang.Thread.run(Thread.java:662)
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
    at sun.nio.ch.IOUtil.read(IOUtil.java:171)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
    at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:60)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:242)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:178)
    at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
    at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
    at java.lang.Thread.run(Thread.java:662)


Thanks,

Bhavesh