HI All
We are using Ignite 2.8.1 and carrying the endurance test lasting for 7 to
12 hours.
Test ran for almost 6 hours and all of a sudden clients got disconnected and
seeing the below logs
what could be the reason for this behavior, we have enough resources like
RAM, CPU during that time
[2021-06-03 00:08:08,172][WARN ][tcp-disco-msg-worker-[8761dfbe
10.119.10.63:47500]-#2][root] Possible failure suppressed accordingly to a
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=grid-timeout-worker,
igniteInstanceName=null, finished=false, heartbeatTs=1622664488170]]]
class org.apache.ignite.IgniteException: GridWorker
[name=grid-timeout-worker, igniteInstanceName=null, finished=false,
heartbeatTs=1622664488170]
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2858)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7759)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
[2021-06-03 00:08:08,172][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:45160]
[2021-06-03 00:08:08,172][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:35382]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:60720]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:54156]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:55260]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:32804]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:54822]
[2021-06-03 00:08:08,175][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:60692]
[2021-06-03 00:08:08,175][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:45316]
Thread=[name=auth-#47, id=92], state=WAITING
Locked pages = []
Locked pages log: name=auth-#47 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=checkpoint-runner-#65, id=112], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#65 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=checkpoint-runner-#66, id=113], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#66 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=checkpoint-runner-#67, id=114], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#67 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=checkpoint-runner-#68, id=115], state=WAITING
Locked pages = []
Thread=[name=checkpoint-runner-#68, id=115], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#68 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#72, id=123], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#72 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#73, id=124], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#73 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#74, id=125], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#74 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#75, id=126], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#75 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#76, id=127], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=client-connector-#76 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#77, id=128], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#77 time=(1622664488172, 2021-06-03
00:08:08.172)
Thread=[name=client-connector-#78, id=129], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#78 time=(1622664488172, 2021-06-03
00:08:08.172)
[2021-06-03 00:08:08,200][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:59602]
[2021-06-03 00:08:08,200][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:2777]
[2021-06-03 00:08:08,200][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:60204]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:37974]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:59656]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:42268]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:50748]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:60798]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:40992]
[2021-06-03 00:08:08,201][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=240000, remoteAddr=/10.129.4.13:37972]
These is our configuration looks like
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="socketWriteTimeout" value="60000"/>
</bean>
</property>
<property name="failureDetectionTimeout" value="60000" />
<property name="clientFailureDetectionTimeout" value="240000" />
<property name="networkTimeout" value="60000" />
<property name="failoverSpi">
<bean
class="org.apache.ignite.spi.failover.always.AlwaysFailoverSpi">
<property name="maximumFailoverAttempts" value="5"/>
</bean>
</property>
<property name="clientConnectorConfiguration">
<bean
class="org.apache.ignite.configuration.ClientConnectorConfiguration">
<property name="sslEnabled" value="true"/>
<property name="handshakeTimeout" value="240000" />
</bean>
</property>
<property name="loadBalancingSpi">
<bean
class="org.apache.ignite.spi.loadbalancing.roundrobin.RoundRobinLoadBalancingSpi">
<property name="perTask" value="true"/>
</bean>
</property>
Rest of the numbers like system, checkpoint threadpool etc are defaults only
WOuld be great if there are pointers towards this issue
Thanks
Naveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
