Hi Phil, One thing I notice with your config is that the cluster.node.protol.port and the zookeeper ports are the same - these should not be the same. Node.protocol.port is used by NiFi cluster to communicate between nodes, the zookeeper.connect.string port should be the port that zookeeper service is listening on. The zookeeper port is configured by the clientPort property in the zookeeper.properties file. This would make your connect string: 'nifi.zookeeper.connect.string=nifi1.domain:2180,nifi2.domain:2180', where 2180 is whatever clientPort is configured.
You can read more about how NiFi uses Zookeeper and how to configure it here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#state_management. Let us know what happens once these properties are configured correctly. Nathan On 9/30/18, 11:07 PM, "Phil H" <[email protected]> wrote: Hi guys, Pulling my hair out trying to solve my Zookeeper problems. I have two 1.6.0 servers that I am trying to cluster. Here is the except from the properties files – all other properties are default so omitted for clarity. The servers are set up to run HTTPS, and the interface works via the browser, so I believe the certificates are correctly installed. Server nifi1.domain: nifi.cluster.is.node=true nifi.cluster.node.address=nifi1.domain nifi.cluster.node.protocol.port=10000 nifi.zookeeper.connect.string=nifi2.domain:10000,nifi1.domain:10000 nifi.zookeeper.root.node=/nifi Server nifi2.domain: nifi.cluster.is.node=true nifi.cluster.node.address=nifi2.domain nifi.cluster.node.protocol.port=10000 nifi.zookeeper.connect.string=nifi1.domain:10000,nifi2.domain:10000 nifi.zookeeper.root.node=/nifi I am getting these errors (this is from server 2, but seeing the same on server 1 apart from a different address, of course): 2018-10-01 20:54:16,332 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 10000 2018-10-01 20:54:16,381 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow 2018-10-01 20:54:16,435 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node: nifi2.domain:443 2018-10-01 20:54:16,769 ERROR [Process Cluster Protocol Request-1] o.a.nifi.security.util.CertificateUtils The incoming request did not contain client certificates and thus the DN cannot be extracted. Check that the other endpoint is providing a complete client certificate chain 2018-10-01 20:54:16,771 WARN [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Failed processing protocol message from nifi2 due to org.apache.nifi.cluster.protocol.ProtocolException: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated org.apache.nifi.cluster.protocol.ProtocolException: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:225) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.dispatchRequest(SocketProtocolListener.java:131) at org.apache.nifi.io.socket.SocketListener$2$1.run(SocketListener.java:136) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.security.cert.CertificateException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:314) at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromSSLSocket(CertificateUtils.java:269) at org.apache.nifi.cluster.protocol.impl.SocketProtocolListener.getRequestorDN(SocketProtocolListener.java:223) ... 5 common frames omitted Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:440) at org.apache.nifi.security.util.CertificateUtils.extractPeerDNFromClientSSLSocket(CertificateUtils.java:299) ... 7 common frames omitted 2018-10-01 20:54:32,249 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2018-10-01 20:54:32,250 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
