Hello all,
I'm working on 3 nodes cluster, sometimes nodes get disconnected and there is
no way to connect them again using the Cluster menu.
I checked the logs and this is what is reported, can you explain what is going
on ?
AP
2016-11-22 11:17:41,943 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new session at /10.1.23.73:38018
2016-11-22 11:17:41,946 INFO [CommitProcessor:2] o.a.zookeeper.server.ZooKeeperServer Established session 0x25876659c2a11d4 with negotiated timeout 4000 for client /10.1.23.73:38018
2016-11-22 11:17:41,946 INFO [Heartbeat Monitor Thread-1-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-22 11:17:46,957 INFO [Heartbeat Monitor Thread-1-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-22 11:17:46,997 INFO [Process Cluster Protocol Request-10] o.a.n.c.c.node.NodeClusterCoordinator Status of mid1-a128-2.buongiorno.com:8080 changed from NodeConnectionStatus[nodeId=mid1-a128-2.buongiorno.com:8080, state=DISCONNECTED, Disconnect Code=Node Failed to Startup Properly, Disconnect Reason=org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: java.lang.ClassCastException, updateId=63] to NodeConnectionStatus[nodeId=mid1-a128-2.buongiorno.com:8080, state=CONNECTING, updateId=64]
2016-11-22 11:17:47,002 INFO [Process Cluster Protocol Request-10-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-22 11:17:47,013 INFO [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 5ac744f9-0c70-4a09-baac-05587e86429d (type=NODE_STATUS_CHANGE, length=1082 bytes) from mid1-a128-3.buongiorno.com in 117 millis
2016-11-22 11:17:47,676 INFO [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Processing reconnection request from manager.
2016-11-22 11:17:47,676 INFO [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Finished processing request f4a7b0b0-eb37-4dd8-a46e-0668f12a6bf9 (type=RECONNECTION_REQUEST, length=1048576 bytes) from mid1-a128-2.buongiorno.com:8080 in 455 millis
2016-11-22 11:17:47,676 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator Resetting cluster node statuses from {mid1-a128-2.buongiorno.com:8080=NodeConnectionStatus[nodeId=mid1-a128-2.buongiorno.com:8080, state=CONNECTING, updateId=64], mid1-a128-1.buongiorno.com:8080=NodeConnectionStatus[nodeId=mid1-a128-1.buongiorno.com:8080, state=CONNECTED, updateId=61], mid1-a128-3.buongiorno.com:8080=NodeConnectionStatus[nodeId=mid1-a128-3.buongiorno.com:8080, state=CONNECTED, updateId=60]} to {mid1-a128-2.buongiorno.com:8080=NodeConnectionStatus[nodeId=mid1-a128-2.buongiorno.com:8080, state=CONNECTING, updateId=64], mid1-a128-1.buongiorno.com:8080=NodeConnectionStatus[nodeId=mid1-a128-1.buongiorno.com:8080, state=CONNECTED, updateId=61], mid1-a128-3.buongiorno.com:8080=NodeConnectionStatus[nodeId=mid1-a128-3.buongiorno.com:8080, state=CONNECTED, updateId=60]}
2016-11-22 11:17:48,391 ERROR [Reconnect to Cluster] o.a.nifi.controller.StandardFlowService Handling reconnection request failed due to: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: java.lang.ClassCastException
org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: java.lang.ClassCastException
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:886) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.apache.nifi.controller.StandardFlowService.handleReconnectionRequest(StandardFlowService.java:592) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.apache.nifi.controller.StandardFlowService.access$300(StandardFlowService.java:97) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.apache.nifi.controller.StandardFlowService$2.run(StandardFlowService.java:404) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.lang.ClassCastException: null
2016-11-22 11:17:48,392 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator mid1-a128-2.buongiorno.com:8080 requested disconnection from cluster due to org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: java.lang.ClassCastException
2016-11-22 11:17:48,392 INFO [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator Status of mid1-a128-2.buongiorno.com:8080 changed from NodeConnectionStatus[nodeId=mid1-a128-2.buongiorno.com:8080, state=CONNECTING, updateId=64] to NodeConnectionStatus[nodeId=mid1-a128-2.buongiorno.com:8080, state=DISCONNECTED, Disconnect Code=Node Failed to Startup Properly, Disconnect Reason=org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: java.lang.ClassCastException, updateId=64]
2016-11-22 11:17:48,394 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new session at /10.1.23.73:38024
2016-11-22 11:17:48,396 INFO [CommitProcessor:2] o.a.zookeeper.server.ZooKeeperServer Established session 0x25876659c2a11d5 with negotiated timeout 4000 for client /10.1.23.73:38024
2016-11-22 11:17:48,396 INFO [Reconnect to Cluster-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-22 11:17:48,405 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] o.a.zookeeper.server.ZooKeeperServer Client attempting to establish new session at /10.1.23.73:38026
2016-11-22 11:17:48,407 INFO [CommitProcessor:2] o.a.zookeeper.server.ZooKeeperServer Established session 0x25876659c2a11d6 with negotiated timeout 4000 for client /10.1.23.73:38026
2016-11-22 11:17:48,407 INFO [Reconnect to Cluster-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-11-22 11:17:48,516 ERROR [Reconnect to Cluster] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for mid1-a128-2.buongiorno.com:8080 -- Node disconnected from cluster due to org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: java.lang.ClassCastException
2016-11-22 11:17:51,348 INFO [Provenance Maintenance Thread-1] o.a.n.p.lucene.UpdateMinimumEventId Updated Minimum Event ID for Provenance Event Repository - Minimum Event ID now 327527
2016-11-22 11:17:51,348 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.lucene.UpdateMinimumEventId@2e56b63d on Provenance Event file /mnt/provenance_repo/327526.prov.gz in 301844 nanos
2016-11-22 11:17:51,348 INFO [Provenance Maintenance Thread-1] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Event file /mnt/provenance_repo/327526.prov.gz
2016-11-22 11:17:51,349 INFO [Provenance Maintenance Thread-1] o.a.n.p.lucene.DeleteIndexAction Removed expired Provenance Table-of-Contents file /mnt/provenance_repo/toc/327526.toc
2016-11-22 11:17:51,349 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully performed Expiration Action org.apache.nifi.provenance.expiration.FileRemovalAction@3c51a1aa on Provenance Event file /mnt/provenance_repo/327526.prov.gz in 116284 nanos
2016-11-22 11:17:51,968 INFO [Heartbeat Monitor Thread-1-EventThread] o.a.c.f.state.ConnectionStateManager
