Hi Our 3 node Zookeeper ensemble got powered down, and upon powering up the zookeeper could get quorum and kept throwing these errors. As a result our Kafka cluster was unusable. What is the best way to revive ZK cluster in such situations ? Please suggest.
2018-08-17_00:59:18.87009 2018-08-17 00:59:18,869 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot open channel to 2 at election address /1.1.1.143:3888 2018-08-17_00:59:18.87011 java.net.ConnectException: Connection refused 2018-08-17_00:59:18.87011 at java.net.PlainSocketImpl.socketConnect(Native Method) 2018-08-17_00:59:18.87011 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) 2018-08-17_00:59:18.87012 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 2018-08-17_00:59:18.87012 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 2018-08-17_00:59:18.87013 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 2018-08-17_00:59:18.87013 at java.net.Socket.connect(Socket.java:589) 2018-08-17_00:59:18.87013 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) 2018-08-17_00:59:18.87014 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) 2018-08-17_00:59:18.87014 at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) 2018-08-17_00:59:18.87014 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) 2018-08-17_00:59:18.87034 2018-08-17 00:59:18,870 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@184] - Resolved hostname: 1.1.1.143 to address: /1.1.1.143 2018-08-17_00:59:18.87095 2018-08-17 00:59:18,870 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot open channel to 3 at election address /1.1.1.144:3888 2018-08-17_00:59:18.87097 java.net.ConnectException: Connection refused 2018-08-17_00:59:18.87097 at java.net.PlainSocketImpl.socketConnect(Native Method) 2018-08-17_00:59:18.87097 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) 2018-08-17_00:59:18.87098 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 2018-08-17_00:59:18.87098 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 2018-08-17_00:59:18.87098 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 2018-08-17_00:59:18.87098 at java.net.Socket.connect(Socket.java:589) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) Thanks. R
