Looks like some non-Ignite application connects to the Ignite server, then sends unexpected data or disconnects quickly.
Could it be some kind of a security tool, port scanner, or a misconfigured service somewhere on the network? On Mon, Sep 16, 2024 at 3:59 AM MJ <6733...@qq.com> wrote: > Hi Igniters, > > > > I am experiencing the “Failed to process selector key” error once every > one or two days. Every time it appears received and rejected multiple > communication connections and then threw the exception. > > Below logging is about “Broken pipe” original exception but not only > “Broken pipe”, occasionally the “Failed to process selector key” wraps > “Connection Reset”, “javax.net.ssl.SSLException: Failed to encrypt data > (SSL engine error) [status=CLOSED, handshakeStatus=NOT_HANDSHAKING”. > > > > Is there any solution to fix it ? or its configuration can be improved ? > > > > Ignite 2.16.0 / 4 data nodes, running in openshift 4 > > > > ---- config of communicationSpi > > <property name="communicationSpi"> > > <bean > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> > > <!-- <property name="localPort" > value="48100" /> --> > > <property name="messageQueueLimit" > value="1024"/> > > <property name="socketWriteTimeout" > value="25000"/> > > <property name="idleConnectionTimeout" > value="60000"/> > > </bean> > > </property> > > > > > > 24-09-15 17:18:35.146 [INFO ] > grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication > connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35160] > > 24-09-15 17:18:35.147 [INFO ] > grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when > already connected to this node, rejecting > [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, > rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] > > 24-09-15 17:18:35.357 [INFO ] > grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication > connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35162] > > 24-09-15 17:18:35.358 [INFO ] > grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when > already connected to this node, rejecting > [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, > rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] > > 24-09-15 17:18:35.568 [INFO ] > grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication > connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35164] > > 24-09-15 17:18:35.569 [INFO ] > grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when > already connected to this node, rejecting > [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, > rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] > > 24-09-15 17:18:35.975 [ERROR] > grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi% > o.a.i.s.c.t.TcpCommunicationSpi:137 - Failed to process selector key > [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker > [super=AbstractNioClientWorker [idx=1, bytesRcvd=29406013584, bytesSent=0, > bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker > [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi, > finished=false, heartbeatTs=1726435114873, hashCode=1144648384, > interrupted=false, > runner=grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%]]], > writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], > inRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0, > rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551, > nodeLeft=false, node=TcpDiscoveryNode > [id=7c28b6bc-8991-47a2-b69c-6adba0482713, consistentId=10.254.13.83, > 127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 127.0.0.1], > sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], > discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, > loc=false, ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], > connected=false, connectCnt=205, queueLimit=131072, reserveCnt=260, > pairedConnections=false], outRecovery=GridNioRecoveryDescriptor > [acked=20129536, resendCnt=0, rcvCnt=19533551, sentCnt=20129879, > reserved=true, lastAck=19533551, nodeLeft=false, node=TcpDiscoveryNode > [id=7c28b6bc-8991-47a2-b69c-6adba0482713, consistentId=10.254.13.83, > 127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 127.0.0.1], > sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], > discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, > loc=false, ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], > connected=false, connectCnt=205, queueLimit=131072, reserveCnt=260, > pairedConnections=false], closeSocket=true, > outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1, > super=GridNioSessionImpl [locAddr=/10.254.32.162:52542, rmtAddr=/ > 10.254.13.83:47100, createTime=1726435114863, closeTime=0, > bytesSent=164200, bytesRcvd=468, bytesSent0=0, bytesRcvd0=0, > sndSchedTime=1726435114863, lastSndTime=1726435114972, > lastRcvTime=1726435114972, readsPaused=false, > filterChain=FilterChain[filters=[GridNioCodecFilter > [parser=o.a.i.i.util.nio.GridDirectParser@5196c6f7, directMode=true], > GridConnectionBytesVerifyFilter, SSL filter], accepted=false, > markedForClose=true]]] java.io.IOException: Broken pipe > > at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native > Method) ~[?:?] > > at > java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[?:?] > > at > java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132) ~[?:?] > > at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76) ~[?:?] > > at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53) ~[?:?] > > at > java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532) > ~[?:?] > > at > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWriteSsl(GridNioServer.java:1524) > ~[ignite-core-2.16.0.jar:2.16.0] > > at > org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1405) > ~[ignite-core-2.16.0.jar:2.16.0] > > at > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2530) > [ignite-core-2.16.0.jar:2.16.0] > > at > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2282) > [ignite-core-2.16.0.jar:2.16.0] > > at > org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1911) > [ignite-core-2.16.0.jar:2.16.0] > > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) > [ignite-core-2.16.0.jar:2.16.0] > > at java.base/java.lang.Thread.run(Thread.java:842) [?:?] > > > > > > > > Thanks, > > -MJ >
