Hi all,
experiencing with Kafka on Kubernetes I have the following error on Kafka
server reconnection ...
A cluster with one zookeeper and two kafka server ... I turn off the zookeeper
pod but kubernetes restart it and guaratees the same IP address for it but the
kafka server starts to retry connection failing with following trace :
[2016-05-10 15:40:55,046] WARN Session 0x1549b308dd20002 for server
10.0.0.184/10.0.0.184:2181, unexpected error, closing socket connection and
attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2016-05-10 15:40:55,149] INFO zookeeper state changed (Disconnected)
(org.I0Itec.zkclient.ZkClient)
[2016-05-10 15:40:57,093] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:57,093] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:57,158] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:58,936] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:58,936] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:58,937] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:00,845] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:00,845] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:00,846] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:02,071] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:02,071] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:02,072] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:03,336] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:03,336] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:03,337] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:05,121] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:05,121] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:05,122] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
You can see when the first zookeeper dies and connection is lost ... and all
the retries by kafka server in order to connect to the new one (same IP, same
port).
Why the zookeeper server closes the connection (I can see FIN ACK frames on
Wireshark)
Thanks,
Paolo.
Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor
Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience
