Hello Igniters:
When I started 20 nodes on one machine , i got a Exception like this "Failed
to find security context for subject with given ID :
a7e071b3-de48-3ec1-9d24-de6cbe6c7bf1".....And then it looks like something is
blocking .
Has anyone encountered this problem?
PS：Ignite version：2.11.0
OS：CentOS
Huty
2021/11/12
java.lang.IllegalStateException: Failed to find security context for subject
with given ID : a7e071b3-de48-3ec1-9d24-de6cbe6c7bf1
at
org.apache.ignite.internal.processors.security.IgniteSecurityProcessor.withContext(IgniteSecurityProcessor.java:153)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1907)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
[17:41:50,340][SEVERE][sys-stripe-5-#6][] Critical system error detected. Will
be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Failed to
find security context for subject with given ID :
a7e071b3-de48-3ec1-9d24-de6cbe6c7bf1]]
java.lang.IllegalStateException: Failed to find security context for subject
with given ID : a7e071b3-de48-3ec1-9d24-de6cbe6c7bf1
at
org.apache.ignite.internal.processors.security.IgniteSecurityProcessor.withContext(IgniteSecurityProcessor.java:153)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1907)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
[17:41:50,342][SEVERE][sys-stripe-5-#6][FailureProcessor] No deadlocked threads
detected.
[17:41:50,422][SEVERE][sys-stripe-5-#6][FailureProcessor] Thread dump at
2021/11/12 17:41:50 GMT+08:00
Thread [name="sys-#198", id=229, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@478a8ddc,
ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="mgmt-#197", id=228, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71439f24,
ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
......
.....
.....
[17:42:00,597][SEVERE][tcp-disco-msg-worker-[f18796d4
0:0:0:0:0:0:0:1%lo:47503]-#3-#77][G] Blocked system-critical thread has been
detected. This can lead to cluster-wide undefined behaviour
[workerName=sys-stripe-1, threadName=sys-stripe-1-#2, blockedFor=10s]
[17:42:00] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-1, igniteInstanceName=null,
finished=false, heartbeatTs=1636710110267]]]
[17:42:12,615][SEVERE][grid-timeout-worker-#38][G] Blocked system-critical
thread has been detected. This can lead to cluster-wide undefined behaviour
[workerName=sys-stripe-1, threadName=sys-stripe-1-#2, blockedFor=22s]
[17:42:12] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-1, igniteInstanceName=null,
finished=false, heartbeatTs=1636710110267]]]
[17:42:21,638][SEVERE][tcp-disco-msg-worker-[f18796d4
0:0:0:0:0:0:0:1%lo:47503]-#3-#77][G] Blocked system-critical thread has been
detected. This can lead to cluster-wide undefined behaviour
[workerName=sys-stripe-1, threadName=sys-stripe-1-#2, blockedFor=31s]
[17:42:21] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-1, igniteInstanceName=null,
finished=false, heartbeatTs=1636710110267]]]
[17:42:33,653][SEVERE][tcp-disco-msg-worker-[f18796d4
0:0:0:0:0:0:0:1%lo:47503]-#3-#77][G] Blocked system-critical thread has been
detected. This can lead to cluster-wide undefined behaviour
[workerName=sys-stripe-1, threadName=sys-stripe-1-#2, blockedFor=43s]
[17:42:33] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=sys-stripe-1, igniteInstanceName=null,
finished=false, heartbeatTs=1636710110267]]]
[17:42:42,669][SEVERE][tcp-disco-msg-worker-[f18796d4
0:0:0:0:0:0:0:1%lo:47503]-#3-#77][G] Blocked system-critical thread has been
detected. This can lead to cluster-wide undefined behaviour
[workerName=sys-stripe-1, threadName=sys-stripe-1-#2, blockedFor=52s]
[17:42:42] Possible failure suppressed accordingly to a configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OP
