RE: Random continuous NetworkException on client and EOFException on server.log

[Deepak Jain]

Hello Everyone,

Alongwith the below exception, also getting in stderr

org.apache.kafka.clients.producer.internals.Sender completeBatch
WARNING: [Producer clientId=producer-1] Received invalid metadata error in 
produce request on partition <topic-name> due to 
org.apache.kafka.common.errors.NetworkException: Disconnected from node 0. 
Going to request metadata update now

-----Original Message-----
From: Deepak Jain <deepak.j...@cumulus-systems.com> 
Sent: 27 January 2022 21:14
To: users@kafka.apache.org
Subject: Random continuous NetworkException on client and EOFException on 
server.log

Hello Everyone,

We are using Kafka 2.8.1 Broker/Client system in our prod env.

Getting following exception randomly after 1 hour or so.

java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.NetworkException: Disconnected from node 0
        at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:98)
        at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:81)
        at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:30)

After enabling debug log on Kafka, found out below EOF exception is also thrown 
almost at same time when the above NetworkException is thrown.

DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Connection with /IP 
disconnected (org.apache.kafka.common.network.Selector)
java.io.EOFException
        at 
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:97)
        at 
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
        at 
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
        at 
org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
        at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at kafka.network.Processor.poll(SocketServer.scala:989)
        at kafka.network.Processor.run(SocketServer.scala:892)
        at java.lang.Thread.run(Thread.java:748)

This exception does not seems to be having any impact on the actual data 
transfer but they are still coming in the logs.

Can anyone please let me know the reason behind them and help me find out the 
root cause for it?

Regards,
Deepak