Hi John, I don't see any GC pressure or STW pauses either. If not GC then it might have been caused by a network glitch or some long-running operation started by the app. These logs statement
[22:26:21,827][WARNING][tcp-disco-client-message-worker-#10%xxxxxx%][TcpDiscoverySpi] Client node considered as unreachable and will be dropped from cluster, because no metrics update messages received in interval: TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be caused by network problems or long GC pause on client node, try to increase this parameter. [nodeId=b07182d0-bf70-4318-9fe3-d7d5228bd6ef, clientFailureDetectionTimeout=30000] [22:26:21,839][WARNING][tcp-disco-client-message-worker-#12%xxxxxx%][TcpDiscoverySpi] Client node considered as unreachable and will be dropped from cluster, because no metrics update messages received in interval: TcpDiscoverySpi.clientFailureDetectionTimeout() ms. It may be caused by network problems or long GC pause on client node, try to increase this parameter. [nodeId=302cff60-b88d-40da-9e12-b955e6bf973d, clientFailureDetectionTimeout=30000] [22:26:21,847][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=partition-exchanger, blockedFor=57s] 22:26:21,954][SEVERE][ttl-cleanup-worker-#48%xxxxxx%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=partition-exchanger, igniteInstanceName=xxxxxx, finished=false, heartbeatTs=1568931981805]]] - Denis On Thu, Oct 3, 2019 at 11:50 AM John Smith <[email protected]> wrote: > So I have been monitoring my node and the same one seems to stop once in a > while. > > https://www.dropbox.com/s/7n5qfsl5uyi1obt/ignite-logs.zip?dl=0 > > I have attached the GC logs and the ignite logs. From what I see from > gc.logs I don't see big pauses. I could be wrong. > > The machine is 16GB and I have the configs here: > https://www.dropbox.com/s/hkv38s3vce5a4sk/ignite-config.xml?dl=0 > > Here are the JVM settings... > > if [ -z "$JVM_OPTS" ] ; then > JVM_OPTS="-Xms2g -Xmx2g -server -XX:MaxMetaspaceSize=256m" > fi > > JVM_OPTS="$JVM_OPTS -XX:+UseG1GC -verbose:gc -XX:+PrintGCDetails > -Xloggc:/var/log/apache-ignite/gc.log" > > JVM_OPTS="${JVM_OPTS} -Xss16m" >
