Hi Igniters,
I am experiencing the “Failed to process selector key” error once every one or two days. Every time it appears received and rejected multiple communication connections and then threw the exception. Below logging is about “Broken pipe” original exception but not only “Broken pipe”, occasionally the “Failed to process selector key” wraps “Connection Reset”, “javax.net.ssl.SSLException: Failed to encrypt data (SSL engine error) [status=CLOSED, handshakeStatus=NOT_HANDSHAKING”. Is there any solution to fix it ? or its configuration can be improved ? Ignite 2.16.0 / 4 data nodes, running in openshift 4 ---- config of communicationSpi <property name="communicationSpi"> <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> <!-- <property name="localPort" value="48100" /> --> <property name="messageQueueLimit" value="1024"/> <property name="socketWriteTimeout" value="25000"/> <property name="idleConnectionTimeout" value="60000"/> </bean> </property> 24-09-15 17:18:35.146 [INFO ] grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35160] 24-09-15 17:18:35.147 [INFO ] grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already connected to this node, rejecting [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] 24-09-15 17:18:35.357 [INFO ] grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35162] 24-09-15 17:18:35.358 [INFO ] grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already connected to this node, rejecting [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] 24-09-15 17:18:35.568 [INFO ] grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35164] 24-09-15 17:18:35.569 [INFO ] grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already connected to this node, rejecting [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] 24-09-15 17:18:35.975 [ERROR] grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi% o.a.i.s.c.t.TcpCommunicationSpi:137 - Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=29406013584, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi, finished=false, heartbeatTs=1726435114873, hashCode=1144648384, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0, rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551, nodeLeft=false, node=TcpDiscoveryNode [id=7c28b6bc-8991-47a2-b69c-6adba0482713, consistentId=10.254.13.83,127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 127.0.0.1], sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, loc=false, ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], connected=false, connectCnt=205, queueLimit=131072, reserveCnt=260, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0, rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551, nodeLeft=false, node=TcpDiscoveryNode [id=7c28b6bc-8991-47a2-b69c-6adba0482713, consistentId=10.254.13.83,127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 127.0.0.1], sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, loc=false, ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], connected=false, connectCnt=205, queueLimit=131072, reserveCnt=260, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1, super=GridNioSessionImpl [locAddr=/10.254.32.162:52542, rmtAddr=/10.254.13.83:47100, createTime=1726435114863, closeTime=0, bytesSent=164200, bytesRcvd=468, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1726435114863, lastSndTime=1726435114972, lastRcvTime=1726435114972, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@5196c6f7, directMode=true], GridConnectionBytesVerifyFilter, SSL filter], accepted=false, markedForClose=true]]] java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?] at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[?:?] at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132) ~[?:?] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76) ~[?:?] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53) ~[?:?] at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532) ~[?:?] at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWriteSsl(GridNioServer.java:1524) ~[ignite-core-2.16.0.jar:2.16.0] at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1405) ~[ignite-core-2.16.0.jar:2.16.0] at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2530) [ignite-core-2.16.0.jar:2.16.0] at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2282) [ignite-core-2.16.0.jar:2.16.0] at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1911) [ignite-core-2.16.0.jar:2.16.0] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) [ignite-core-2.16.0.jar:2.16.0] at java.base/java.lang.Thread.run(Thread.java:842) [?:?] Thanks, -MJ
logs.txt
Description: Binary data
