Re: rejecting communication connection & Failed to process selector key

Thu, 19 Sep 2024 23:24:25 -0700 

Yes - kubernetes discovery spi.   No client nodes are out of the 
cluster.


and below setting for communication spi.



---- config of communicationSpi

<property name="communicationSpi"&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <!-- <property name="localPort" value="48100" /&gt; --&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <property name="messageQueueLimit" value="1024"/&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <property name="socketWriteTimeout" value="25000"/&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <property name="idleConnectionTimeout" value="60000"/&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 </bean&gt;

</property&gt;

&nbsp;






---Original---
From: "Humphrey Lopez"<hmmlo...@gmail.com&gt;
Date: Fri, Sep 20, 2024 13:27 PM
To: "user"<user@ignite.apache.org&gt;;
Subject: Re: rejecting communication connection &amp; Failed to process 
selector key


Are you using the kubernetes discovery SPI?
Humphrey&nbsp;

On 16 Sep 2024, at 02:58, MJ <6733...@qq.com&gt; wrote:


Hi Igniters,



 
&nbsp;
 
I am experiencing the “Failed to process selector key” error once every one or 
two days. Every time it appears received and rejected multiple communication 
connections and then threw the exception.
 
Below logging is about “Broken pipe” original exception but not only “Broken 
pipe”, occasionally the “Failed to process selector key” wraps “Connection 
Reset”, “javax.net.ssl.SSLException:  Failed to encrypt data (SSL engine error) 
[status=CLOSED, handshakeStatus=NOT_HANDSHAKING”. &nbsp;
 
&nbsp;
 
Is there any solution to fix it ? or its configuration can be improved ?
 
&nbsp;
 
Ignite 2.16.0 / 4 data nodes, running in openshift 4 
 
&nbsp;
 
---- config of communicationSpi
 
<property name="communicationSpi"&gt;
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"&gt;
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <!-- <property name="localPort" value="48100" /&gt; --&gt;
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <property name="messageQueueLimit" value="1024"/&gt;
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <property name="socketWriteTimeout" value="25000"/&gt;
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 <property name="idleConnectionTimeout" value="60000"/&gt;
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 </bean&gt;
 
</property&gt;
 
&nbsp;
 
&nbsp;
 
24-09-15 17:18:35.146 [INFO ] 
grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication 
connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35160]&nbsp; 
 
24-09-15 17:18:35.147 [INFO ] 
grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already 
connected to this node, rejecting 
[locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40,  
rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713]&nbsp; 
 
24-09-15 17:18:35.357 [INFO ] 
grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication 
connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35162]&nbsp; 
 
24-09-15 17:18:35.358 [INFO ] 
grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already 
connected to this node, rejecting 
[locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40,  
rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713]&nbsp; 
 
24-09-15 17:18:35.568 [INFO ] 
grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:117 - Accepted incoming communication 
connection [locAddr=/10.254.32.162:47100, rmtAddr=/10.254.13.83:35164]&nbsp; 
 
24-09-15 17:18:35.569 [INFO ] 
grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:117 - Received incoming connection when already 
connected to this node, rejecting 
[locNode=52437bc3-3dfe-4f76-bec6-d2f22f8a5d40,  
rmtNode=7c28b6bc-8991-47a2-b69c-6adba0482713]&nbsp; 
 
24-09-15 17:18:35.975 [ERROR] 
grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi% 
o.a.i.s.c.t.TcpCommunicationSpi:137 - Failed to process selector key 
[ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker 
[super=AbstractNioClientWorker  [idx=1, bytesRcvd=29406013584, bytesSent=0, 
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker 
[name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi, 
finished=false, heartbeatTs=1726435114873, hashCode=1144648384, 
interrupted=false,  
runner=grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%]]], 
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], 
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], 
inRecovery=GridNioRecoveryDescriptor [acked=20129536, resendCnt=0, 
rcvCnt=19533551,  sentCnt=20129879, reserved=true, lastAck=19533551, 
nodeLeft=false, node=TcpDiscoveryNode [id=7c28b6bc-8991-47a2-b69c-6adba0482713, 
consistentId=10.254.13.83,127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 
127.0.0.1], sockAddrs=HashSet [/10.254.13.83:47500,  /127.0.0.1:47500], 
discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, loc=false, 
ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], connected=false, 
connectCnt=205, queueLimit=131072, reserveCnt=260, pairedConnections=false], 
outRecovery=GridNioRecoveryDescriptor  [acked=20129536, resendCnt=0, 
rcvCnt=19533551, sentCnt=20129879, reserved=true, lastAck=19533551, 
nodeLeft=false, node=TcpDiscoveryNode [id=7c28b6bc-8991-47a2-b69c-6adba0482713, 
consistentId=10.254.13.83,127.0.0.1:47500, addrs=ArrayList [10.254.13.83, 
127.0.0.1],  sockAddrs=HashSet [/10.254.13.83:47500, /127.0.0.1:47500], 
discPort=47500, order=3, intOrder=3, lastExchangeTime=1724822271382, loc=false, 
ver=2.16.0#20231215-sha1:7bde6a42, isClient=false], connected=false, 
connectCnt=205, queueLimit=131072, reserveCnt=260,  pairedConnections=false], 
closeSocket=true,  
outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1,
 super=GridNioSessionImpl [locAddr=/10.254.32.162:52542, 
rmtAddr=/10.254.13.83:47100, createTime=1726435114863, closeTime=0, 
bytesSent=164200, bytesRcvd=468, bytesSent0=0,  bytesRcvd0=0, 
sndSchedTime=1726435114863, lastSndTime=1726435114972, 
lastRcvTime=1726435114972, readsPaused=false, 
filterChain=FilterChain[filters=[GridNioCodecFilter 
[parser=o.a.i.i.util.nio.GridDirectParser@5196c6f7, directMode=true], 
GridConnectionBytesVerifyFilter,  SSL filter], accepted=false, 
markedForClose=true]]] java.io.IOException: Broken pipe
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[?:?]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132) ~[?:?]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:76) ~[?:?]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:53) ~[?:?]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:532) 
~[?:?]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at 
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWriteSsl(GridNioServer.java:1524)
 ~[ignite-core-2.16.0.jar:2.16.0]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at 
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1405)
 ~[ignite-core-2.16.0.jar:2.16.0]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2530)
 [ignite-core-2.16.0.jar:2.16.0]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2282)
 [ignite-core-2.16.0.jar:2.16.0]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1911)
 [ignite-core-2.16.0.jar:2.16.0]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) 
[ignite-core-2.16.0.jar:2.16.0]
 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 at java.base/java.lang.Thread.run(Thread.java:842) [?:?]
 
&nbsp;
 
&nbsp;
 
&nbsp;
 
Thanks,
 
-MJ
 
 
<logs.txt&gt;

Reply via email to