[jira] [Commented] (IGNITE-20023) Unstable cluster topology leads to flaky test

[Roman Puchkovskiy (Jira)]

    [ 
https://issues.apache.org/jira/browse/IGNITE-20023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750370#comment-17750370
 ] 

Roman Puchkovskiy commented on IGNITE-20023:
--------------------------------------------

What I was able to find in the logs:

# There is a lot of noise about one node not being let by other nodes to the 
topology as it has already left (this seems to be irrelevant to the test 
failure)
# There is an NPE (that probably caused the test failure), but additional 
information is needed to find the culprit. I'm adding it in the attached PR
# There is a secondary NPE (it is suppressed), I'm fixing it in the PR

> Unstable cluster topology leads to flaky test
> ---------------------------------------------
>
>                 Key: IGNITE-20023
>                 URL: https://issues.apache.org/jira/browse/IGNITE-20023
>             Project: Ignite
>          Issue Type: Bug
>    Affects Versions: 3.0
>            Reporter: Alexey Scherbakov
>            Assignee: Roman Puchkovskiy
>            Priority: Major
>              Labels: ignite-3
>
> {noformat}
> 2023-07-21 18:30:15:108 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] >>> Starting test: 
> ItTxDistributedTestThreeNodesThreeReplicas#testTxClosureAsync, displayName: 
> testTxClosureAsync(), workDir: 
> D:\work\ignite-3\modules\table\build\work\ItTxDistributedTestThreeNodesThreeReplicas\ttca_25996
> 2023-07-21 18:30:15:142 +0300 [INFO][main][ConnectionManager] Server started 
> [address=/0:0:0:0:0:0:0:0:20001]
> 2023-07-21 18:30:15:142 +0300 
> [INFO][ForkJoinPool.commonPool-worker-7][ConnectionManager] Server started 
> [address=/0:0:0:0:0:0:0:0:20000]
> 2023-07-21 18:30:15:142 +0300 
> [INFO][ForkJoinPool.commonPool-worker-3][ConnectionManager] Server started 
> [address=/0:0:0:0:0:0:0:0:20002]
> 2023-07-21 18:30:15:145 +0300 
> [INFO][itdttntr_n_20000-srv-worker-1][RecoveryServerHandshakeManager] Failed 
> to acquire recovery descriptor during handshake, it is held by: [id: 
> 0x348087cd, L:/192.168.0.138:63056 - R:/192.168.0.138:20001]
> 2023-07-21 18:30:15:147 +0300 
> [INFO][itdttntr_n_20002-client-1][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x8017c1b2, 
> L:/192.168.0.138:20002 - R:/192.168.0.138:63058]
> 2023-07-21 18:30:15:147 +0300 
> [INFO][itdttntr_n_20002-client-2][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0xb26b9231, 
> L:/192.168.0.138:20002 - R:/192.168.0.138:63059]
> 2023-07-21 18:30:15:149 +0300 
> [INFO][itdttntr_n_20001-client-3][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x3dd8aa9e, 
> L:/192.168.0.138:20001 - R:/192.168.0.138:63056]
> 2023-07-21 18:30:15:149 +0300 
> [INFO][itdttntr_n_20002-client-3][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0xb26b9231, 
> L:/192.168.0.138:20002 - R:/192.168.0.138:63059]
> 2023-07-21 18:30:15:149 +0300 
> [INFO][itdttntr_n_20002-client-4][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x8017c1b2, 
> L:/192.168.0.138:20002 - R:/192.168.0.138:63058]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20002]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20002]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20001]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_20002]]
> 2023-07-21 18:30:15:150 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_20001]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][ForkJoinPool.commonPool-worker-3][ScaleCubeTopologyService] Node 
> joined [node=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][ForkJoinPool.commonPool-worker-3][ScaleCubeTopologyService] Topology 
> snapshot [nodes=[itdttntr_n_20000, itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:151 +0300 [INFO][main][ScaleCubeTopologyService] Node 
> joined [node=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null]]
> 2023-07-21 18:30:15:151 +0300 [INFO][main][ScaleCubeTopologyService] Topology 
> snapshot [nodes=[itdttntr_n_20000, itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][ForkJoinPool.commonPool-worker-7][ScaleCubeTopologyService] Node 
> joined [node=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][ForkJoinPool.commonPool-worker-7][ScaleCubeTopologyService] Topology 
> snapshot [nodes=[itdttntr_n_20000, itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:151 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] The cluster has been 
> started
> 2023-07-21 18:30:15:173 +0300 [INFO][main][ConnectionManager] Server started 
> [address=/0:0:0:0:0:0:0:0:19999]
> 2023-07-21 18:30:15:181 +0300 
> [INFO][itdttntr_n_19999-client-4][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x70cd17f5, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63117]
> 2023-07-21 18:30:15:181 +0300 
> [INFO][itdttntr_n_19999-client-5][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x66c6df09, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63116]
> 2023-07-21 18:30:15:181 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=ace76a7e-4b4e-45aa-b696-22d1813c9a89, 
> name=itdttntr_n_19999, address=192.168.0.138:19999, nodeMetadata=null]]
> 2023-07-21 18:30:15:181 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002, 
> itdttntr_n_20001]]
> 2023-07-21 18:30:15:181 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=ace76a7e-4b4e-45aa-b696-22d1813c9a89, 
> name=itdttntr_n_19999, address=192.168.0.138:19999, nodeMetadata=null]]
> 2023-07-21 18:30:15:181 +0300 
> [INFO][itdttntr_n_19999-client-6][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0xccdddbe8, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63118]
> 2023-07-21 18:30:15:182 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002, 
> itdttntr_n_20001]]
> 2023-07-21 18:30:15:182 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=ace76a7e-4b4e-45aa-b696-22d1813c9a89, 
> name=itdttntr_n_19999, address=192.168.0.138:19999, nodeMetadata=null]]
> 2023-07-21 18:30:15:182 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002, 
> itdttntr_n_20001]]
> 2023-07-21 18:30:15:183 +0300 
> [INFO][itdttntr_n_19999-client-7][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x70cd17f5, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63117]
> 2023-07-21 18:30:15:183 +0300 
> [INFO][itdttntr_n_19999-client-1][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0xccdddbe8, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63118]
> 2023-07-21 18:30:15:183 +0300 
> [INFO][itdttntr_n_19999-client-8][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x66c6df09, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63116]
> 2023-07-21 18:30:15:185 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null]]
> 2023-07-21 18:30:15:185 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20002]]
> 2023-07-21 18:30:15:185 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null]]
> 2023-07-21 18:30:15:185 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:185 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Node joined 
> [node=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null]]
> 2023-07-21 18:30:15:185 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:186 +0300 
> [INFO][itdttntr_n_19999-client-2][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x66c6df09, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63116]
> 2023-07-21 18:30:15:186 +0300 [INFO][main][ScaleCubeTopologyService] Node 
> joined [node=ClusterNode [id=ace76a7e-4b4e-45aa-b696-22d1813c9a89, 
> name=itdttntr_n_19999, address=192.168.0.138:19999, nodeMetadata=null]]
> 2023-07-21 18:30:15:186 +0300 
> [INFO][itdttntr_n_19999-client-3][RecoveryClientHandshakeManager] Failed to 
> acquire recovery descriptor during handshake, it is held by: [id: 0x70cd17f5, 
> L:/192.168.0.138:19999 - R:/192.168.0.138:63117]
> 2023-07-21 18:30:15:186 +0300 [INFO][main][ScaleCubeTopologyService] Topology 
> snapshot [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002, 
> itdttntr_n_20001]]
> 2023-07-21 18:30:15:186 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] Replica manager has 
> been started, node=[ClusterNode [id=ace76a7e-4b4e-45aa-b696-22d1813c9a89, 
> name=itdttntr_n_19999, address=192.168.0.138:19999, nodeMetadata=null]]
> 2023-07-21 18:30:15:186 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] The client has been 
> started
> 2023-07-21 18:30:15:234 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] Replica manager has 
> been started, node=[ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null]]
> 2023-07-21 18:30:15:263 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] Replica manager has 
> been started, node=[ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null]]
> 2023-07-21 18:30:15:320 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] Replica manager has 
> been started, node=[ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null]]
> 2023-07-21 18:30:15:320 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] Raft servers have 
> been started
> 2023-07-21 18:30:15:321 +0300 [INFO][main][Loza] Start new raft 
> node=RaftNodeId [groupId=1_part_0, peer=Peer [consistentId=itdttntr_n_20001, 
> idx=0]] with initial configuration=PeersAndLearners [peers=SetN [Peer 
> [consistentId=itdttntr_n_20002, idx=0], Peer [consistentId=itdttntr_n_20001, 
> idx=0], Peer [consistentId=itdttntr_n_20000, idx=0]], learners=SetN []]
> 2023-07-21 18:30:15:321 +0300 [INFO][main][FSMCallerImpl] Starts FSMCaller 
> successfully.
> 2023-07-21 18:30:15:322 +0300 [WARNING][main][LocalSnapshotStorage] No data 
> for snapshot reader 
> build\work\ItTxDistributedTestThreeNodesThreeReplicas\ttca_25996\node0\1_part_0_itdttntr_n_20001_0\snapshot.
> 2023-07-21 18:30:15:322 +0300 [INFO][main][NodeImpl] Node 
> <1_part_0/itdttntr_n_20001> init, term=0, lastLogId=LogId [index=0, term=0], 
> conf=itdttntr_n_20002,itdttntr_n_20001,itdttntr_n_20000, oldConf=.
> 2023-07-21 18:30:15:322 +0300 [INFO][main][RaftGroupService] Start the 
> RaftGroupService successfully <1_part_0/itdttntr_n_20001>
> 2023-07-21 18:30:15:324 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:324 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:324 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002]]
> 2023-07-21 18:30:15:324 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002]]
> 2023-07-21 18:30:15:324 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=bee1e04c-e2f0-48c0-ad54-eefdd6928278, 
> name=itdttntr_n_20001, address=192.168.0.138:20001, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:324 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999, itdttntr_n_20002]]
> 2023-07-21 18:30:15:327 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-8][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:327 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-8][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:fefb2389-82c9-4118-ba31-ecd239f51632 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:334 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:334 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999]]
> 2023-07-21 18:30:15:335 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-1][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:335 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-1][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:71eb3899-3a4a-46ff-877e-9cba5644e114 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-20000-105][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_20000, itdttntr_n_19999]]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-19999-108][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_19999]]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-20002-106][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_19999, itdttntr_n_20002]]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=496bbe6d-66b7-4aa9-9807-e1dd5207d900, 
> name=itdttntr_n_20000, address=192.168.0.138:20000, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:335 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_19999, itdttntr_n_20002, itdttntr_n_20001]]
> 2023-07-21 18:30:15:336 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-1][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:336 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-1][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:4d59a697-e0c1-4b6c-bca2-2c844a4cfaba Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:337 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-2][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:337 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-2][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:16075a90-93b8-4118-9541-0c8c0682d764 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:337 +0300 
> [WARNING][itdttntr_n_20002-srv-worker-6][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:337 +0300 
> [ERROR][itdttntr_n_20002-srv-worker-6][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:37bf20f6-af97-4250-8d2f-2ccb04992d55 Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:345 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-3][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:345 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-3][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:822c6d0c-09b1-4210-89e5-bd7e5e159d1c Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:348 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-3][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:348 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-3][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:aa98ae37-ac77-4f49-b308-c6bb90e4cdb3 Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:348 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Node left 
> [member=ClusterNode [id=ade225ca-5ef0-4b03-9dbe-edbe2e2ba928, 
> name=itdttntr_n_20002, address=192.168.0.138:20002, nodeMetadata=null], 
> eventType=LEAVING]
> 2023-07-21 18:30:15:348 +0300 
> [INFO][sc-cluster-20001-107][ScaleCubeTopologyService] Topology snapshot 
> [nodes=[itdttntr_n_19999, itdttntr_n_20001]]
> 2023-07-21 18:30:15:349 +0300 
> [WARNING][itdttntr_n_20002-srv-worker-7][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:349 +0300 
> [ERROR][itdttntr_n_20002-srv-worker-7][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:a739a92b-eaba-4090-8d6a-7a3707ca6479 Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:349 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-4][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:349 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-4][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:2da41592-ca40-4a75-bbd2-ad76d7bf5d9a Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:354 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-5][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:354 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-5][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:60edfa5b-39d4-48fd-876f-1d0bb44c034c Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:356 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-7][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:356 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-7][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:325f3c89-932a-47b0-bfdc-65fa2fc50e32 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:357 +0300 
> [WARNING][itdttntr_n_20002-srv-worker-8][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:357 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-7][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:358 +0300 
> [ERROR][itdttntr_n_20002-srv-worker-8][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:8f5d0327-87b0-472c-8581-9e524ef717ee Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:358 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-7][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:52f8f59f-d539-4af0-9a6a-22923e2452b1 Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:364 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-8][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:365 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-8][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:7756fa82-c91e-4a18-940c-a3bdcf5c83d7 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:367 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-1][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:367 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-1][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:5a1fb248-1088-40fd-890f-115df19e11b6 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:367 +0300 
> [WARNING][itdttntr_n_20002-srv-worker-1][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:367 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-1][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:367 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-1][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:b50cfb39-23cf-49f6-8c7e-1ab9819ad06c Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:367 +0300 
> [ERROR][itdttntr_n_20002-srv-worker-1][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:bc4290a8-62be-4ea9-a97c-95f61e81ad18 Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:368 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-2][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:368 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-2][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:3ce026a1-f9c8-41d5-9b62-db27eb69c8bd Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:375 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-3][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:375 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-3][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:0c1a4063-e4d6-4557-a545-aad1591df920 Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:375 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-2][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:375 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-2][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:4f43885c-b06c-4a08-ba48-56cce0acce40 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:377 +0300 
> [WARNING][itdttntr_n_20002-srv-worker-2][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:377 +0300 
> [ERROR][itdttntr_n_20002-srv-worker-2][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:0b7ed0da-695b-4631-97da-888941269dd5 Handshake rejected by client: 
> itdttntr_n_20002:ade225ca-5ef0-4b03-9dbe-edbe2e2ba928 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:378 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-5][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:378 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-5][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:fb71454e-6a68-4cce-8b72-d57b5a51b51a Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:385 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-7][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:386 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-7][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:138a5dc2-7fd8-42ab-89ac-81c36edd34ed Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:387 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-5][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:387 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-5][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:e62ab22c-45ac-4b70-85f5-ce43d95b6758 Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:387 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-8][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:387 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-8][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:26874a49-ede7-4b74-8273-9bd7a0f74f75 Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:395 +0300 
> [WARNING][itdttntr_n_20001-srv-worker-6][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:395 +0300 
> [ERROR][itdttntr_n_20001-srv-worker-6][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:ceb10214-a533-4697-aa15-5b1e1a74674e Handshake rejected by client: 
> itdttntr_n_20001:bee1e04c-e2f0-48c0-ad54-eefdd6928278 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:396 +0300 [INFO][sc-cluster-20001-107][Transport] 
> Stopping [address=192.168.0.138:20001]
> 2023-07-21 18:30:15:396 +0300 [INFO][sc-cluster-20001-107][Transport] Stopped 
> [address=192.168.0.138:20001]
> 2023-07-21 18:30:15:397 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-2][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:397 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-2][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:750a1dd6-52c3-4f34-9c24-171e7db3d51e Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:399 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-3][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:399 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-3][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:468dd39d-c371-40ad-8ea5-560ca09576a3 Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:405 +0300 [INFO][sc-cluster-20002-106][Transport] 
> Stopping [address=192.168.0.138:20002]
> 2023-07-21 18:30:15:405 +0300 [INFO][sc-cluster-20002-106][Transport] Stopped 
> [address=192.168.0.138:20002]
> 2023-07-21 18:30:15:405 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-4][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:405 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-4][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:dbc5665b-99b1-4388-80e5-3a77285e1bec Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:15:406 +0300 [INFO][sc-cluster-20000-105][Transport] 
> Stopping [address=192.168.0.138:20000]
> 2023-07-21 18:30:15:406 +0300 [INFO][sc-cluster-20000-105][Transport] Stopped 
> [address=192.168.0.138:20000]
> 2023-07-21 18:30:15:407 +0300 
> [WARNING][itdttntr_n_20000-srv-worker-5][RecoveryServerHandshakeManager] 
> Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
> 2023-07-21 18:30:15:407 +0300 
> [ERROR][itdttntr_n_20000-srv-worker-5][FailureHandler] Critical failure
> org.apache.ignite.lang.IgniteException: IGN-CMN-65535 
> TraceId:b189117d-7c61-450f-b3fb-c1cf35b3b96b Handshake rejected by client: 
> itdttntr_n_20000:496bbe6d-66b7-4aa9-9807-e1dd5207d900 is stale, server should 
> be restarted so that clients can connect
>       at 
> org.apache.ignite.internal.network.recovery.RecoveryServerHandshakeManager.onMessage(RecoveryServerHandshakeManager.java:185)
>       at 
> org.apache.ignite.internal.network.netty.HandshakeHandler.channelRead(HandshakeHandler.java:92)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
>       at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
>       at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
>       at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
>       at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>       at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
>       at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
>       at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
>       at 
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
>       at 
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>       at 
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>       at java.base/java.lang.Thread.run(Thread.java:834)
> 2023-07-21 18:30:16:506 +0300 [INFO][sc-cluster-19999-108][Transport] 
> Stopping [address=192.168.0.138:19999]
> 2023-07-21 18:30:16:506 +0300 [INFO][sc-cluster-19999-108][Transport] Stopped 
> [address=192.168.0.138:19999]
> 2023-07-21 18:30:17:395 +0300 
> [INFO][%itdttntr_n_20001%JRaft-ElectionTimer-0][NodeImpl] Node 
> <1_part_0/itdttntr_n_20001> term 0 start preVote.
> 2023-07-21 18:30:17:437 +0300 [INFO][main][Loza] Stop raft node=RaftNodeId 
> [groupId=1_part_0, peer=Peer [consistentId=itdttntr_n_20001, idx=0]]
> 2023-07-21 18:30:17:437 +0300 [INFO][main][NodeImpl] Node 
> <1_part_0/itdttntr_n_20001> shutdown, currTerm=0 state=STATE_FOLLOWER.
> 2023-07-21 18:30:17:446 +0300 [INFO][main][LocalRaftMetaStorage] Save raft 
> meta, 
> path=build\work\ItTxDistributedTestThreeNodesThreeReplicas\ttca_25996\node0\1_part_0_itdttntr_n_20001_0\meta,
>  term=0, votedFor=, cost time=9 ms
> 2023-07-21 18:30:17:447 +0300 [INFO][main][FSMCallerImpl] Shutting down 
> FSMCaller...
> 2023-07-21 18:30:17:448 +0300 [INFO][main][RaftGroupService] Stop the 
> RaftGroupService successfully.
> 2023-07-21 18:30:17:449 +0300 
> [INFO][%itdttntr_n_20001%JRaft-Common-Executor-1][LogThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20001%JRaft-Common-Executor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@668113c1[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 4].
> 2023-07-21 18:30:17:460 +0300 
> [INFO][%itdttntr_n_20001%JRaft-Node-Scheduler-1][LogScheduledThreadPoolExecutor]
>  ThreadPool is terminated: %itdttntr_n_20001%JRaft-Node-Scheduler-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@6103e586[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 2].
> 2023-07-21 18:30:17:460 +0300 
> [INFO][%itdttntr_n_20001%JRaft-ElectionTimer-0][LogScheduledThreadPoolExecutor]
>  ThreadPool is terminated: %itdttntr_n_20001%JRaft-ElectionTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@2f659c14[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 1].
> 2023-07-21 18:30:17:460 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20001%JRaft-VoteTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@472bfea7[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:461 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20001%JRaft-StepDownTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@6f4da521[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:461 +0300 
> [INFO][%itdttntr_n_20001%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor]
>  ThreadPool is terminated: %itdttntr_n_20001%JRaft-SnapshotTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@3994880e[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:461 +0300 
> [INFO][%itdttntr_n_20001%JRaft-Response-Processor-1][LogThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20001%JRaft-Response-Processor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@78b07f51[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 2].
> 2023-07-21 18:30:17:461 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20001%JRaft-Request-Processor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@23829f80[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:467 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20002%JRaft-Common-Executor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@3e5e859d[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20002%JRaft-Node-Scheduler-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@26989a37[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20002%JRaft-ElectionTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@24fe5768[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20002%JRaft-VoteTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@13b7b2e8[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20002%JRaft-StepDownTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@590faf6e[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20002%JRaft-SnapshotTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@3ba4df5a[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20002%JRaft-Response-Processor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@6d378a51[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:475 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20002%JRaft-Request-Processor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@43b945ed[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:479 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20000%JRaft-Common-Executor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@6769e62b[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:487 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20000%JRaft-Node-Scheduler-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@7a2aab93[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:488 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20000%JRaft-ElectionTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@2b1ed385[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:488 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20000%JRaft-VoteTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@22017d26[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:488 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20000%JRaft-StepDownTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@af19ff3[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:488 +0300 [INFO][main][LogScheduledThreadPoolExecutor] 
> ThreadPool is terminated: %itdttntr_n_20000%JRaft-SnapshotTimer-, 
> org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@2528cb0d[Shutting
>  down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:488 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20000%JRaft-Response-Processor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@78f7b6d6[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:488 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool 
> is terminated: %itdttntr_n_20000%JRaft-Request-Processor-, 
> org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@51361ca3[Shutting 
> down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
> 0].
> 2023-07-21 18:30:17:493 +0300 
> [INFO][main][ItTxDistributedTestThreeNodesThreeReplicas] >>> Stopping test: 
> ItTxDistributedTestThreeNodesThreeReplicas#testTxClosureAsync, displayName: 
> testTxClosureAsync(), cost: 2386ms.
> java.lang.NullPointerException
>       at 
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.randomNode(RaftGroupServiceImpl.java:705)
>       at 
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.randomNode(RaftGroupServiceImpl.java:670)
>       at 
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.refreshLeader(RaftGroupServiceImpl.java:224)
>       at 
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.start(RaftGroupServiceImpl.java:180)
>       at 
> org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.start(TopologyAwareRaftGroupService.java:182)
>       at 
> org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory.startRaftGroupService(TopologyAwareRaftGroupServiceFactory.java:71)
>       at 
> org.apache.ignite.internal.raft.Loza.startRaftGroupNodeInternal(Loza.java:350)
>       at 
> org.apache.ignite.internal.raft.Loza.startRaftGroupNode(Loza.java:230)
>       at 
> org.apache.ignite.internal.raft.Loza.startRaftGroupNode(Loza.java:183)
>       at 
> org.apache.ignite.distributed.ItTxDistributedTestSingleNode.startTable(ItTxDistributedTestSingleNode.java:487)
>       at 
> org.apache.ignite.distributed.ItTxDistributedTestSingleNode.before(ItTxDistributedTestSingleNode.java:355)
>       at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>       at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>       at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>       at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>       at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>       at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>       at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
>       at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeEachMethod(TimeoutExtension.java:78)
>       at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>       at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>       at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
>       at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>       at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
>       at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
>       at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
>       at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
>       at 
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
>       at 
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeBeforeEachMethodAdapter$23(ClassBasedTestDescriptor.java:505)
>       at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachMethods$3(TestMethodTestDescriptor.java:174)
>       at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:202)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:202)
>       at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachMethods(TestMethodTestDescriptor.java:171)
>       at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:134)
>       at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
>       at 
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
>       at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
>       at 
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
>       at 
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
>       at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
>       at 
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
>       at 
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
>       at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
>       at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
>       at 
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
>       at 
> org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
>       at 
> org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
>       at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
>       at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
>       at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
>       at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
>       at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
>       at 
> org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
>       at 
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
>       at 
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
>       at 
> org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
>       at 
> org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
>       at 
> com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
>       at 
> com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
>       at 
> com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
>       at 
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
>       at 
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
>       at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
>       Suppressed: java.lang.NullPointerException
>               at 
> org.apache.ignite.distributed.ItTxDistributedTestSingleNode.after(ItTxDistributedTestSingleNode.java:659)
>               at 
> org.apache.ignite.distributed.ItTxDistributedTestThreeNodesThreeReplicas.after(ItTxDistributedTestThreeNodesThreeReplicas.java:59)
>               at 
> jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>               at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>               at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>               at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
>               at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>               at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>               at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>               at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
>               at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:110)
>               at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>               at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>               at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
>               at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>               at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
>               at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
>               at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
>               at 
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
>               at 
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:520)
>               at 
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$24(ClassBasedTestDescriptor.java:510)
>               at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$10(TestMethodTestDescriptor.java:243)
>               at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:276)
>               at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>               at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$14(TestMethodTestDescriptor.java:276)
>               at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
>               at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:275)
>               at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:241)
>               at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:142)
>               ... 48 more
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)