I suspect your openshift networking is doing something wrong: NAT is particularly suspicious.
Share your discovery configuration and openshift network layout. On Mon, Sep 16, 2024 at 4:38 AM MJ <6733...@qq.com> wrote: > Donot think so. As below the remote ip 10.254.13.83 > <http://10.254.13.83:47100/> is the another server node. > --- log > Accepted incoming communication connection [locAddr=/10.254.32.162:47100, > rmtAddr=/10.254.13.83:35160] > super=GridNioSessionImpl [locAddr=/10.254.32.162:52542, rmtAddr=/ > 10.254.13.83:47100 > --- > > so the multiple connections kept being rejected were between two server > nodes. Any scenarios could cause that ? it appears to be that the original > connection was shutdown or interrupted fast by one node but the other node > was not aware of the connection close event, or was not informed ? Any > configuration can help on that ? > > > Thanks, > -MJ > > Original Email > > From:"Pavel Tupitsyn"< ptupit...@apache.org >; > > Sent Time:2024/9/16 12:58 > > To:"user"< user@ignite.apache.org >; > > Subject:Re: rejecting communication connection & Failed to process > selector key > > Looks like some non-Ignite application connects to the Ignite server, then > sends unexpected data or disconnects quickly. > > Could it be some kind of a security tool, port scanner, or a misconfigured > service somewhere on the network? > > On Mon, Sep 16, 2024 at 3:59 AM MJ <6733...@qq.com> wrote: > >> Hi Igniters, >> >> >> >> I am experiencing the “Failed to process selector key” error once every >> one or two days. Every time it appears received and rejected multiple >> communication connections and then threw the exception. >> >> Below logging is about “Broken pipe” original exception but not only >> “Broken pipe”, occasionally the “Failed to process selector key” wraps >> “Connection Reset”, “javax.net.ssl.SSLException: Failed to encrypt data >> (SSL engine error) [status=CLOSED, handshakeStatus=NOT_HANDSHAKING”. >> >> >> >> Is there any solution to fix it ? or its configuration can be improved ? >> >> >> >> Ignite 2.16.0 / 4 data nodes, running in openshift 4 >> >> >> >> ---- config of communicationSpi >> >> <property name="communicationSpi"> >> >> <bean >> class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> >> >> <!-- <property name="localPort" >> value="48100" /> --> >> >> <property name="messageQueueLimit" >> value="1024"/> >> >> <property name="socketWriteTimeout" >> value="25000"/> >> >> <property name="idleConnectionTimeout" >> value="60000"/> >> >> </bean> >> >> </property> >> >> >> >> >> >> 24-09-15 17:18:35.146 [INFO ] >> grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication >> connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35160] >> >> 24-09-15 17:18:35.147 [INFO ] >> grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when >> already connected to this node, rejecting >> [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, >> rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] >> >> 24-09-15 17:18:35.357 [INFO ] >> grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication >> connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35162] >> >> 24-09-15 17:18:35.358 [INFO ] >> grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when >> already connected to this node, rejecting >> [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, >> rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] >> >> 24-09-15 17:18:35.568 [INFO ] >> grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication >> connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35164] >> >> 24-09-15 17:18:35.569 [INFO ] >> grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when >> already connected to this node, rejecting >> [locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40, >> rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713] >> >> 24-09-15 17:18:35.975 [ERROR] >> grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi% >> o.a.i.s.c.t.TcpCommunicationSpi:137 - Failed to process selector key >> [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker >> [super=AbstractNioClientWorker [idx=1, bytesRcvd=29406013584, bytesSent=0, >> bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker >> [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi, >> finished=false, heartbeatTs=1726435114873, hashCode=1144648384, >> interrupted=false, >> runner=grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%]]], >> writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], >> readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], >> inRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0, >> rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551, >> nodeLeft=false, node=TcpDiscoveryNode >> [id=7c28b6bc-8991-47a2-b69c-6adba0482713, consistentId=10.254.13.83, >> 127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 127.0.0.1], >> sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], >> discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, >> loc=false, ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], >> connected=false, connectCnt=205, queueLimit=131072, reserveCnt=260, >> pairedConnections=false], outRecovery=GridNioRecoveryDescriptor >> [acked=20129536, resendCnt=0, rcvCnt=19533551, sentCnt=20129879, >> reserved=true, lastAck=19533551, nodeLeft=false, node=TcpDiscoveryNode >> [id=7c28b6bc-8991-47a2-b69c-6adba0482713, consistentId=10.254.13.83, >> 127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 127.0.0.1], >> sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], >> discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, >> loc=false, ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], >> connected=false, connectCnt=205, queueLimit=131072, reserveCnt=260, >> pairedConnections=false], closeSocket=true, >> outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1, >> super=GridNioSessionImpl [locAddr=/10.254.32.162:52542, rmtAddr=/ >> 10.254.13.83:47100, createTime=1726435114863, closeTime=0, >> bytesSent=164200, bytesRcvd=468, bytesSent0=0, bytesRcvd0=0, >> sndSchedTime=1726435114863, lastSndTime=1726435114972, >> lastRcvTime=1726435114972, readsPaused=false, >> filterChain=FilterChain[filters=[GridNioCodecFilter >> [parser=o.a.i.i.util.nio.GridDirectParser@5196c6f7, directMode=true], >> GridConnectionBytesVerifyFilter, SSL filter], accepted=false, >> markedForClose=true]]] java.io.IOException: Broken pipe >> >> at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native >> Method) ~[?:?] >> >> at >> java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[?:?] >> >> at >> java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132) ~[?:?] >> >> at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76) >> ~[?:?] >> >> at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53) >> ~[?:?] >> >> at >> java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532) >> ~[?:?] >> >> at >> org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWriteSsl(GridNioServer.java:1524) >> ~[ignite-core-2.16.0.jar:2.16.0] >> >> at >> org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1405) >> ~[ignite-core-2.16.0.jar:2.16.0] >> >> at >> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2530) >> [ignite-core-2.16.0.jar:2.16.0] >> >> at >> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2282) >> [ignite-core-2.16.0.jar:2.16.0] >> >> at >> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1911) >> [ignite-core-2.16.0.jar:2.16.0] >> >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) >> [ignite-core-2.16.0.jar:2.16.0] >> >> at java.base/java.lang.Thread.run(Thread.java:842) [?:?] >> >> >> >> >> >> >> >> Thanks, >> >> -MJ >> >
