[
https://issues.apache.org/jira/browse/ZOOKEEPER-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147853#comment-17147853
]
Brittany Barnes commented on ZOOKEEPER-3828:
--------------------------------------------
Having this same issue with 3.6.1! We're also deploying in docker containers.
This happens to any node that I restart - leader, follower, or observer.
It's breaking all failover functionality and anytime a ZK is restarted it
appears healthy but will not accept any connections and will not rejoin the
quorum. Seeing the same logs when I try to connect:
{code:java}
/usr/bin/java
Connecting to localhost:2181
2020-06-26 20:09:46,413 [myid:] - INFO [main:Environment@98] - Client
environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b,
built on 04/21/2020 15:01 GMT
2020-06-26 20:09:46,418 [myid:] - INFO [main:Environment@98] - Client
environment:host.name=zoo1
2020-06-26 20:09:46,418 [myid:] - INFO [main:Environment@98] - Client
environment:java.version=1.8.0_252
2020-06-26 20:09:46,430 [myid:] - INFO [main:Environment@98] - Client
environment:java.vendor=Oracle Corporation
2020-06-26 20:09:46,430 [myid:] - INFO [main:Environment@98] - Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
2020-06-26 20:09:46,431 [myid:] - INFO [main:Environment@98] - Client
environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf:
2020-06-26 20:09:46,431 [myid:] - INFO [main:Environment@98] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-06-26 20:09:46,431 [myid:] - INFO [main:Environment@98] - Client
environment:java.io.tmpdir=/tmp
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:java.compiler=<NA>
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:os.name=Linux
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:os.arch=amd64
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:os.version=3.10.0-327.28.3.el7.x86_64
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:user.name=root
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:user.home=/root
2020-06-26 20:09:46,434 [myid:] - INFO [main:Environment@98] - Client
environment:user.dir=/apache-zookeeper-3.6.1-bin
2020-06-26 20:09:46,435 [myid:] - INFO [main:Environment@98] - Client
environment:os.memory.free=51MB
2020-06-26 20:09:46,437 [myid:] - INFO [main:Environment@98] - Client
environment:os.memory.max=228MB
2020-06-26 20:09:46,437 [myid:] - INFO [main:Environment@98] - Client
environment:os.memory.total=57MB
2020-06-26 20:09:46,448 [myid:] - INFO [main:ZooKeeper@1005] - Initiating
client connection, connectString=localhost:2181 sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5e265ba4
2020-06-26 20:09:46,459 [myid:] - INFO [main:X509Util@77] - Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS
renegotiation
2020-06-26 20:09:46,483 [myid:] - INFO [main:ClientCnxnSocket@239] -
jute.maxbuffer value is 1048575 Bytes
2020-06-26 20:09:46,509 [myid:] - INFO [main:ClientCnxn@1703] -
zookeeper.request.timeout value is 0. feature enabled=false
Welcome to ZooKeeper!
2020-06-26 20:09:46,523 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
connection to server localhost/0:0:0:0:0:0:0:1:2181.
2020-06-26 20:09:46,524 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2020-06-26 20:09:46,733 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket connection
established, initiating session, client: /0:0:0:0:0:0:0:1:59158, server:
localhost/0:0:0:0:0:0:0:1:2181
2020-06-26 20:09:46,757 [myid:localhost:2181] - WARN
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0 for
sever localhost/0:0:0:0:0:0:0:1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid 0x0,
likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
[zk: localhost:2181(CONNECTING) 0] 2020-06-26 20:09:47,990
[myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
connection to server localhost/0:0:0:0:0:0:0:1:2181.
2020-06-26 20:09:47,991 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
2020-06-26 20:09:47,991 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket connection
established, initiating session, client: /0:0:0:0:0:0:0:1:59160, server:
localhost/0:0:0:0:0:0:0:1:2181
2020-06-26 20:09:47,997 [myid:localhost:2181] - WARN
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0 for
sever localhost/0:0:0:0:0:0:0:1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid 0x0,
likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
2020-06-26 20:09:49,550 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
connection to server localhost/0:0:0:0:0:0:0:1:2181.
2020-06-26 20:09:49,550 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
2020-06-26 20:09:49,550 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket connection
established, initiating session, client: /0:0:0:0:0:0:0:1:59162, server:
localhost/0:0:0:0:0:0:0:1:2181
2020-06-26 20:09:49,558 [myid:localhost:2181] - WARN
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0 for
sever localhost/0:0:0:0:0:0:0:1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid 0x0,
likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
2020-06-26 20:09:51,624 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
connection to server localhost/0:0:0:0:0:0:0:1:2181.
2020-06-26 20:09:51,624 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
status: Will not attempt to authenticate using SASL (unknown error)
2020-06-26 20:09:51,626 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket connection
established, initiating session, client: /0:0:0:0:0:0:0:1:59164, server:
localhost/0:0:0:0:0:0:0:1:2181
2020-06-26 20:09:51,653 [myid:localhost:2181] - WARN
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0 for
sever localhost/0:0:0:0:0:0:0:1:2181, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException.
EndOfStreamException: Unable to read additional data from server sessionid 0x0,
likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262)
{code}
> zookeeper CLI client gets connection timeout when the leader node is restarted
> ------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-3828
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3828
> Project: ZooKeeper
> Issue Type: Bug
> Components: java client
> Affects Versions: 3.6.1
> Reporter: Aishwarya Soni
> Priority: Minor
> Attachments: debug_logs.zip, node1.txt, node2.txt, node3.txt,
> node4.txt, node5.txt
>
>
> I have configured 5 nodes zookeeper cluster using 3.6.1 version in a docker
> containerized environment. As a part of some destructive testing, I restarted
> zookeeper leader. Now, re-election happened and all 5 nodes (containers) are
> back in good state with new leader. But when I login to one of the container
> and go inside zk Cli (./zkCli.sh) and run the cmd *ls /* I see below error,
> {color:#000000} {color}
> *{color:#000000}[zk: localhost:2181(CONNECTING) 1]{color}*
> *{color:#000000}[zk: localhost:2181(CONNECTING) 1] ls /{color}*
> *{color:#000000}2020-05-14 23:48:26,556 [myid:localhost:2181] - WARN
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1229] - Client session
> timed out, have not heard from server in 30001ms for session id 0x0{color}*
> *{color:#000000}2020-05-14 23:48:26,556 [myid:localhost:2181] - WARN
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0
> for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting
> reconnect except it is a SessionExpiredException.{color}*
> *{color:#000000}org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> Client session timed out, have not heard from server in 30001ms for session
> id 0x0{color}*
> *{color:#000000}at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1230){color}*
> *{color:#000000}KeeperErrorCode = ConnectionLoss for /{color}*
> *{color:#000000}[zk: localhost:2181(CONNECTING) 2] 2020-05-14 23:48:28,089
> [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
> connection to server localhost/127.0.0.1:2181.{color}*
> *{color:#000000}2020-05-14 23:48:28,089 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
> status: Will not attempt to authenticate using SASL (unknown error){color}*
> *{color:#000000}2020-05-14 23:48:28,090 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket
> connection established, initiating session, client: /127.0.0.1:60384, server:
> localhost/127.0.0.1:2181{color}*
> *{color:#000000}2020-05-14 23:48:58,119 [myid:localhost:2181] - WARN
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1229] - Client session
> timed out, have not heard from server in 30030ms for session id 0x0{color}*
> *{color:#000000}2020-05-14 23:48:58,120 [myid:localhost:2181] - WARN
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0
> for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting
> reconnect except it is a SessionExpiredException.{color}*
> *{color:#000000}org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> Client session timed out, have not heard from server in 30030ms for session
> id 0x0{color}*
> *{color:#000000}at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1230){color}*
> *{color:#000000}2020-05-14 23:49:00,003 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
> connection to server localhost/127.0.0.1:2181.{color}*
> *{color:#000000}2020-05-14 23:49:00,004 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
> status: Will not attempt to authenticate using SASL (unknown error){color}*
> *{color:#000000}2020-05-14 23:49:00,004 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket
> connection established, initiating session, client: /127.0.0.1:32936, server:
> localhost/127.0.0.1:2181{color}*
> *{color:#000000}2020-05-14 23:49:30,032 [myid:localhost:2181] - WARN
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1229] - Client session
> timed out, have not heard from server in 30029ms for session id 0x0{color}*
> *{color:#000000}2020-05-14 23:49:30,033 [myid:localhost:2181] - WARN
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0
> for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting
> reconnect except it is a SessionExpiredException.{color}*
> *{color:#000000}org.apache.zookeeper.ClientCnxn$SessionTimeoutException:
> Client session timed out, have not heard from server in 30029ms for session
> id 0x0{color}*
> *{color:#000000}at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1230){color}*
> *{color:#000000}2020-05-14 23:49:31,230 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket
> connection to server localhost/127.0.0.1:2181.{color}*
> *{color:#000000}2020-05-14 23:49:31,230 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config
> status: Will not attempt to authenticate using SASL (unknown error){color}*
> *{color:#000000}2020-05-14 23:49:31,230 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket
> connection established, initiating session, client: /127.0.0.1:33766, server:
> localhost/127.0.0.1:2181{color}*
> {color:#000000}Does anyone know what could possibly be wrong? For reference:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2164{color}
> This behavior is observed on all the nodes when the leader is restarted. All
> is good when a follower is restarted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
