But if it's the client node that's failing why is the server node stopping? I'm pretty sure we do verry simple put and get operations. All the client nodes are started as client=true
On Thu., Oct. 3, 2019, 4:18 p.m. Denis Magda, <[email protected]> wrote: > Hi John, > > I don't see any GC pressure or STW pauses either. If not GC then it might > have been caused by a network glitch or some long-running operation started > by the app. These logs statement > > > [22:26:21,827][WARNING][tcp-disco-client-message-worker-#10%xxxxxx%][TcpDiscoverySpi] > Client node considered as unreachable and will be dropped from cluster, > because no metrics update messages received in interval: > TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be caused by > network problems or long GC pause on client node, try to increase this > parameter. [nodeId=b07182d0-bf70-4318-9fe3-d7d5228bd6ef, > clientFailureDetectionTimeout=30000] > > > [22:26:21,839][WARNING][tcp-disco-client-message-worker-#12%xxxxxx%][TcpDiscoverySpi] > Client node considered as unreachable and will be dropped from cluster, > because no metrics update messages received in interval: > TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be caused by > network problems or long GC pause on client node, try to increase this > parameter. [nodeId=302cff60-b88d-40da-9e12-b955e6bf973d, > clientFailureDetectionTimeout=30000] > > > [22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][G] Blocked > system-critical thread has been detected. This can lead to cluster-wide > undefined behaviour [threadName=partition-exchanger, blockedFor=57s] > > > 22:26:21,954][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][] Critical system > error detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler > [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext > [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker > [name=partition-exchanger, igniteInstanceName=xxxxxx, finished=false, > heartbeatTs=1568931981805]]] > > > > > - > Denis > > > On Thu, Oct 3, 2019 at 11:50 AM John Smith <[email protected]> wrote: > >> So I have been monitoring my node and the same one seems to stop once in >> a while. >> >> https://www.dropbox.com/s/7n5qfsl5uyi1obt/ignite-logs.zip?dl=0 >> >> I have attached the GC logs and the ignite logs. From what I see from >> gc.logs I don't see big pauses. I could be wrong. >> >> The machine is 16GB and I have the configs here: >> https://www.dropbox.com/s/hkv38s3vce5a4sk/ignite-config.xml?dl=0 >> >> Here are the JVM settings... >> >> if [ -z "$JVM_OPTS" ] ; then >> JVM_OPTS="-Xms2g -Xmx2g -server -XX:MaxMetaspaceSize=256m" >> fi >> >> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC -verbose:gc -XX:+PrintGCDetails >> -Xloggc:/var/log/apache-ignite/gc.log" >> >> JVM_OPTS="${JVM_OPTS} -Xss16m" >> >
