Dmitry Sherstobitov created IGNITE-10943:
--------------------------------------------
Summary: "No next node in topology" infinite messages in log after
cycle cluster nodes restart
Key: IGNITE-10943
URL: https://issues.apache.org/jira/browse/IGNITE-10943
Project: Ignite
Issue Type: Bug
Affects Versions: 2.4
Reporter: Dmitry Sherstobitov
Attachments: grid.1.node.1.jstack.log
Same scenario as in https://issues.apache.org/jira/browse/IGNITE-10878
After cluster restarted here is one node with 100% CPU load and following
messages in log:
{code:java}
2019-01-15T15:16:41,333][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
Message has been added to queue: TcpDiscoveryNodeFailedMessage
[failedNodeId=e006e575-bbc8-4004-8ce3-ddc165d1748c, order=12, warning=null,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=3cfe0715861-24a27aff-e471-4db1-ac46-cda072de17b9, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
2019-01-15T15:16:41,333][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
Pending messages will be resent to local node
2019-01-15T15:16:41,333][INFO ][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP
discovery spawning a new thread for connection [rmtAddr=/172.25.1.40,
rmtPort=59236]
2019-01-15T15:16:41,333][INFO ][tcp-disco-sock-reader-#21][TcpDiscoverySpi]
Started serving remote node connection [rmtAddr=/172.25.1.40:59236,
rmtPort=59236]
2019-01-15T15:16:41,333][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
Message has been added to queue: TcpDiscoveryStatusCheckMessage
[creatorNode=TcpDiscoveryNode [id=24a27aff-e471-4db1-ac46-cda072de17b9,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 172.25.1.40],
sockAddrs=[/172.17.0.1:47500, /0:0:0:0:0:0:0:1%lo:47500,
lab40.gridgain.local/172.25.1.40:47500, /127.0.0.1:47500], discPort=47500,
order=0, intOrder=15, lastExchangeTime=1547554584282, loc=true,
ver=2.4.13#20190114-sha1:a7667ae6, isClient=false], failedNodeId=null,
status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=4cfe0715861-24a27aff-e471-4db1-ac46-cda072de17b9, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
2019-01-15T15:16:41,334][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi]
Ignore message failed nodes, sender node is in fail list
[nodeId=e006e575-bbc8-4004-8ce3-ddc165d1748c,
failedNodes=[a251994d-8df6-4b2d-a28c-18ec55a3a48c,
a5fa9095-2e4b-48e5-803d-551a5ebde558]]
2019-01-15T15:16:41,334][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,334][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,334][DEBUG][tcp-disco-sock-reader-#21][TcpDiscoverySpi]
Initialized connection with remote node
[nodeId=6df245fe-6288-4d93-ab20-2b9ac1b35771, client=false]
2019-01-15T15:16:41,334][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,335][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,335][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,335][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,335][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,335][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,335][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,336][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.
2019-01-15T15:16:41,336][DEBUG][tcp-disco-msg-worker-#2][TcpDiscoverySpi] No
next node in topology.{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
