A communication problem happens between nodes, so it’s difficult to say what the problem is when we only see one side of the story.
Having said that, you either have a networking issue or one of your nodes is seeing long JVM pauses. A long JVM pause means that Ignite code isn’t running, resulting in the node appearing to be dead. It’s usually (but not always) because of Java garbage collection. > On 21 Mar 2023, at 06:11, Abhishek Ubhe <abhishekubhe0...@gmail.com> wrote: > > Hello, > Getting below errors continuously on Ignite node. Please help me with > explaining the below exception when it occurs and why? > > > ERROR org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - Failed to > send message to remote node [node=TcpDiscoveryNode > [id=6cc31b37-7870-413a-b75a-95b4371fc542, > consistentId=mtnclzcom_1679291151812, addrs=ArrayList [10.233.102.12, > 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500 <http://127.0.0.1:47500/>, > /10.233.102.12:47500 <http://10.233.102.12:47500/>, > mtn-kafka-stream-6b74bc74f-6d8jw/13.93.181.185:47500 > <http://13.93.181.185:47500/>], discPort=47500, order=2, intOrder=2, > lastExchangeTime=1679291157038, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, > isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, > ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearTxPrepareRequest > [futId=528e59df681-cef0f6c9-0046-4006-a7c4-9c6514ce6210, miniId=1, > topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], > subjId=6eaca65c-cf03-42d9-8819-31e0825fa86a, taskNameHash=0, txLbl=null, > flags=[implicitSingle], super=GridDistributedTxPrepareRequest [threadId=275, > concurrency=OPTIMISTIC, isolation=READ_COMMITTED, writeVer=GridCacheVersion > [topVer=290770946, order=1679291313990, nodeOrder=1], timeout=0, reads=null, > writes=PredicateCollectionView [IgniteTxEntry [txKey=IgniteTxKey > [key=KeyCacheObjectImpl [part=190, > val=IGNITE_CRUD_ROW_KEY_1234_5678_91011_DLR_CLZ_COM, hasValBytes=true], > cacheId=628281256], val=TxEntryValueHolder [val=com.google.gson.JsonObject > [idHash=85330180, hash=1866240837, members=LinkedHashMap > {ACTIVE_STATUS=com.google.gson.JsonPrimitive [idHash=1159180188, > hash=31429505, value=1], CML_ACCEPTED=com.google.gson.JsonPrimitive > [idHash=1673278339, hash=31429505, value=1], > CML_DATE=com.google.gson.JsonPrimitive [idHash=1909260391, hash=-1099433161, > value=2023-03-20T05:43:23.423]}], op=CREATE], prevVal=TxEntryValueHolder > [val=null, op=NOOP], oldVal=TxEntryValueHolder [val=null, op=NOOP], > entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, > explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], > filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry > [super=GridDistributedCacheEntry [super=GridCacheMapEntry > [key=KeyCacheObjectImpl [part=190, > val=IGNITE_CRUD_ROW_KEY_1234_5678_91011_DLR_CLZ_COM, hasValBytes=true], > val=null, ver=GridCacheVersion [topVer=0, order=0, nodeOrder=0], > hash=-661118938, extras=null, flags=0]]], prepared=0, locked=false, > nodeId=6cc31b37-7870-413a-b75a-95b4371fc542, locMapped=false, expiryPlc=null, > transferExpiryPlc=false, flags=2, partUpdateCntr=0, serReadVer=null, > xidVer=GridCacheVersion [topVer=290770946, order=1679291313990, > nodeOrder=1]]], dhtVers=null, txSize=0, plc=2, txState=null, flags=last, > super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=290770946, > order=1679291313990, nodeOrder=1], committedVers=null, rolledbackVers=null, > cnt=0, super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage > [msgId=120793, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion > [topVer=2, minorTopVer=1], err=null, skipPrepare=false]]]]]]] > org.apache.ignite.IgniteCheckedException: Failed to connect to node due to > unrecoverable exception (is node still alive?). Make sure that each > ComputeTask and cache Transaction has a timeout set in order to prevent > parties from waiting forever in case of network issues > [nodeId=6cc31b37-7870-413a-b75a-95b4371fc542, addrs=[/10.233.102.12:47100 > <http://10.233.102.12:47100/>, > mtn-kafka-stream-6b74bc74f-6d8jw/13.93.181.185:47100 > <http://13.93.181.185:47100/>, /127.0.0.1:47100 <http://127.0.0.1:47100/>], > err= class org.apache.ignite.IgniteCheckedException: Remote node does not > observe current node in topology : 6cc31b37-7870-413a-b75a-95b4371fc542] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:627) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1255) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:689) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:453) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:228) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1186) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1133) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2101) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2196) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1296) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.sendReply(IgniteTxHandler.java:1573) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1327) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:135) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:244) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:242) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569) > ~[ignite-core-2.10.0.jar:2.10.0] > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > ~[ignite-core-2.10.0.jar:2.10.0] > at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332] > Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to > node (is node still alive?). Make sure that each ComputeTask and cache > Transaction has a timeout set in order to prevent parties from waiting > forever in case of network issues > [nodeId=6cc31b37-7870-413a-b75a-95b4371fc542, addrs=[/10.233.102.12:47100 > <http://10.233.102.12:47100/>, > mtn-kafka-stream-6b74bc74f-6d8jw/13.93.181.185:47100 > <http://13.93.181.185:47100/>, /127.0.0.1:47100 <http://127.0.0.1:47100/>]] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:609) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1255) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:689) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:453) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:228) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1186) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1133) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2101) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2196) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1296) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.sendReply(IgniteTxHandler.java:1573) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1327) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:135) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:244) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:242) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55) > ~[ignite-core-2.10.0.jar:2.10.0] > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569) > ~[ignite-core-2.10.0.jar:2.10.0] > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > ~[ignite-core-2.10.0.jar:2.10.0] > at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332] > Caused by: java.net.SocketTimeoutException > at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:129) ~[?:1.8.0_332] > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:465) > ~[ignite-core-2.10.0.jar:2.10.0] > ... 30 more > Caused by: org.apache.ignite.IgniteCheckedException: Remote node does not > observe current node in topology : 6cc31b37-7870-413a-b75a-95b4371fc542 > at > org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:505) > ~[ignite-core-2.10.0.jar:2.10.0] > ... 30 more > -- > Regards, > Abhishek Ubhe >
