Hi , Our setup : Servers - 3 node cluster Reader clients : wait for an update on an entry of a cache ( around 20 of them ) Writer Client : 1
If one of the reader client restarts while the writer is writing into the entry of the cache , the server attempts to send the update to the failed client's local listener . It keeps attempting to communicate with the failed client ( client's continous query local listener ? ) till it timesout as per connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the snippet of the exception below. The complete log is attached as an attachment ) This delays the completion of the transaction that was started by the writer client. Is there any way the writer client could complete the transaction without getting impacted by the reader client restarts ? 2020-10-08 14:35:21,465 [sys-stripe-26-#27] WARN org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Handshake timed out (will stop attempts to perform the handshake) [node=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, connTimeoutStrategy=ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400, currTimeout=600000], err=Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400, currTimeout=600000]], addr=MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103, failureDetectionTimeoutEnabled=true, timeout=0] 2020-10-08 14:35:21,465 [sys-stripe-26-#27] ERROR org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Failed to send message to remote node [node=TcpDiscoveryNode [id=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, consistentId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=ArrayList [0:0:0:0:0:0:0:1, 1.2.3.4, 127.0.0.1], sockAddrs=HashSet [MACHINENAME.COMPANY.LOCAL/1.2.3.4:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0, order=12, intOrder=8, lastExchangeTime=1602163619453, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=true], msg=GridIoMessage [plc=2, topic=T4 [topic=TOPIC_CACHE, id1=94370aa1-e970-37ae-9471-fd583d923522, id2=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, id3=0], topicOrd=-1, ordered=true, timeout=0, skipOnTimeout=true, msg=GridContinuousMessage [type=MSG_EVT_NOTIFICATION, routineId=7dac3ff4-3460-4dc4-8324-a1ebe4561854, data=null, futId=null]]] org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3698) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) ~[ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) ~[ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223742176046300, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582) ~[ignite-core-2.8.1.jar:2.8.1] ... 41 more Suppressed: org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103, MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3700) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:90) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:445) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:443) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1588) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:745) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:606) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:478) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:523) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1077) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:953) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:909) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:123) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:217) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:215) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) ~[ignite-core-2.8.1.jar:2.8.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223772180706400, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582) ~[ignite-core-2.8.1.jar:2.8.1] ... 41 more Caused by: org.apache.ignite.spi.IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy [maxTimeout=600000, totalTimeout=30000, startNanos=223712146125000, currTimeout=600000]] at org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39) ~[ignite-core-2.8.1.jar:2.8.1] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3582) ~[ignite-core-2.8.1.jar:2.8.1] ... 41 more 2020-10-08 14:35:21,467 [sys-stripe-26-#27] ERROR org.apache.ignite.continuous.query [] - Failed to send event notification to node: 3311a67b-bfcb-41af-8c09-b2e8f2fbde9b EXPONENTIALBACKOFFSTRATEGY.txt <http://apache-ignite-users.70518.x6.nabble.com/file/t2757/EXPONENTIALBACKOFFSTRATEGY.txt> -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/