Hello Aleksandr,
First, thank you for taking the time to look into this and for the quick
response. We really appreciate it.
Thanks as well for creating the JIRA ticket; we will keep an eye on it.
As requested, I collected detailed logs for a scenario in which the
deadlock occurred during:
`ignite.reentrantLock(lockName, /* failoverSafe */ true, /* fair */ false,
/* create */ true).tryLock(5, TimeUnit.SECONDS)`
The logs were added manually in the code. For reference, I opened a PR
showing exactly where they were added:
https://github.com/felipeek/ignite/pull/1/changes
The deadlock occurred in the same flow described in my previous message.
The only difference is that, this time, I could not see the
`TOPIC_CONTINUOUS` messages in the logs (unclear why).
Nevertheless, the logs were captured in their entirety and should be
helpful to debug this further. As you can see, an `onNodeRemoved` log is
emitted, followed by a `CAS COMMIT` log, indicating the lock was acquired
after the `nodeRemoved` message was received by the front pod. However, no
subsequent `onUpdate` message is received, causing the front pod to become
stuck in the busy-wait. We believe the `onUpdate` message is not received
because the backend pod does not send it. Under normal conditions, we see a
`CAS COMMIT` log, followed by the `onUpdate` log (which unblocks the front
pod), and finally an `UNLOCK COMMIT` log (when the front unlocks).
Stack trace where the front pod gets stuck indefinitely:
"""at java.lang.Thread.yield([email protected]/Native Method)
at
org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl$Sync.tryAcquire(GridCacheLockImpl.java:401)
at
org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl$Sync.tryAcquire(GridCacheLockImpl.java:432)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos([email protected]
/AbstractQueuedSynchronizer.java:982)
at
org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl.tryLock(GridCacheLockImpl.java:1279)
at app.imdg.ignite.features.map.IgniteImdgMap.lock(IgniteImdgMap.java:252)
at app.imdg.ignite.features.map.IgniteImdgMap.lock(IgniteImdgMap.java:260)
at
app.cfcs.RolesService.loadCertificateRolesMapEntry(RolesService.java:74)"""
Additional important info:
* In the setup where these logs were collected, there were 3 backend pods
(backend-pod-1, backend-pod-2, backend-pod-3) and 1 frontend pod
(front-pod).
* To keep the dataset manageable, I shared only messages emitted by
front-pod and backend-pod-1. front-pod is the one acquiring the lock, while
backend-pod-1 is the pod that left the grid and triggered the issue. The
other two backend pods may appear in some low-level logs but should be less
relevant.
===
Regarding your workaround suggestion to use semaphores: we gave it a try.
When semaphores are used, the "deadlock" frequency decreases significantly.
However, after some tests, we reproduced the same symptom (Ignite being
stuck indefinitely), but with a different (yet seemingly related) stack
trace:
"""at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at java.util.concurrent.locks.LockSupport.park([email protected]
/LockSupport.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:181)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:144)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.close(GridNearTxLocal.java:3991)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.close(GridNearTxLocal.java:3971)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync$1.call(GridCacheSemaphoreImpl.java:325)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync$1.call(GridCacheSemaphoreImpl.java:278)
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.retryTopologySafe(GridCacheUtils.java:1360)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync.compareAndSetGlobalState(GridCacheSemaphoreImpl.java:278)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync.tryReleaseShared(GridCacheSemaphoreImpl.java:245)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared([email protected]
/AbstractQueuedSynchronizer.java:1094)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.release(GridCacheSemaphoreImpl.java:827)
at
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.release(GridCacheSemaphoreImpl.java:815)
at
app.imdg.ignite.features.map.IgniteImdgMap.unlock(IgniteImdgMap.java:288)"""
In this trace, the front-pod gets stuck when releasing the semaphore.
In this test, we used:
`ignite.semaphore(lockName, /* permits */ 1, /* failoverSafe */ true, /*
create */ true).tryAcquire(5, TimeUnit.SECONDS);`
to acquire and
`ignite.semaphore(lockName, /* permits */ 1, /* failoverSafe */ true, /*
create */ true).release()`
to release.
For this reason, at least for now, we decided not to invest further in the
semaphore approach, since it appears to fall into the same class of problem
and could become too convoluted to investigate on two fronts.
We might try next a custom solution using a cache to store custom lock
metadata, together with cache::invoke for acquisition/release of locks with
a custom lease (or something similar, we are evaluating our options).
Regarding IEP-132: that is great news and will certainly help us a lot.
Please let us know if you have any other suggestions or requests.
Thanks,
Felipe
Em ter., 24 de fev. de 2026 às 12:27, Aleksandr Chesnokov <
[email protected]> escreveu:
> Hello Felipe,
>
> Thank you very much for your detailed bug report and investigation.
> Reports like this help us improve Ignite and make it more reliable. Please
> feel free to share any other issues you encounter - your feedback is very
> valuable for the community.
>
> We have reviewed your findings and the code references you provided. We
> agree that this looks like a real liveness issue in IgniteLock. We have
> created a JIRA ticket to track it:
> https://issues.apache.org/jira/browse/IGNITE-27962
>
> We will continue investigating the root cause and possible fixes.
>
> If possible, could you also share debug logs around the topology change
> and lock acquisition (from both the client and the server node that was
> stopped)? In particular, logs covering:
>
> - the transaction commit,
> - continuous query processing,
> - node left / topology change events.
>
> This may help us better understand the race condition and validate a fix.
>
> As a temporary workaround, you may try using IgniteSemaphore(1,
> failoverSafe=true) instead of IgniteLock, if reentrancy is not required in
> your use case.
>
> In addition, we are currently working on improved Rolling Upgrade
> functionality (IEP-132):
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-132+Rolling+Upgrade
> This feature is under active development, and we plan to finalize it in
> upcoming releases. We expect it to improve stability and behavior during
> node restarts and cluster upgrades.
>
> Thank you again for your contribution and detailed analysis.
>
>
> --
> Best regards,
> Aleksandr Chesnokov
>
[2026-02-24T23:59:41.323Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,323",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.293Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,293",
"logLevel": "ERROR",
"threadName": "exchange-worker-#46%front-pod%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsSingleMessage
payload=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null,
partsSizes=null, partHistCntrs=null, err=null, client=true,
exchangeStartTime=1771977581292, finishMsg=null,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=36, minorTopVer=1],
discoEvt=DiscoveryCustomEvent [customMsg=CacheAffinityChangeMessage
[id=58b9e029c91-d3f9f8f0-e12d-46ac-8fa5-19aa1ef93050,
topVer=AffinityTopologyVersion [topVer=36, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true, stopProc=false],
affTopVer=AffinityTopologyVersion [topVer=36, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=1f73012b-df27-44e1-a8a6-9d9e1e101693, consistentId=backend-pod-2,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 100.96.23.12, 127.0.0.1], sockAddrs=null,
discPort=47500, order=1, intOrder=1, loc=false,
ver=2.17.0#20250209-sha1:d53d4540, isClient=false], topVer=36,
msgTemplate=null,
span=org.apache.ignite.internal.processors.tracing.NoopSpan@1eaeb8ac,
nodeId8=1c4499fb, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1771977581284]],
nodeId=1f73012b, evt=DISCOVERY_CUSTOM_EVT], lastVer=GridCacheVersion
[topVer=383456971, order=1771977557076, nodeOrder=5, dataCenterId=0],
super=GridCacheMessage [msgId=13377, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0],
err=null, skipPrepare=false]]]"
}
[2026-02-24T23:59:41.239Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,239",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.231Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,231",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.216Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,216",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.166Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,166",
"logLevel": "ERROR",
"threadName": "qtp1855112877-338",
"sessionID": "nil",
"correlationID": "nil",
"requestID": "nil",
"logCtxID": "84727db0-0bb5-4b93-a9df-f5ebbf1ff19c",
"tenantUser": "nil",
"loggerName":
"org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl",
"message":
"<ignite-sys-atomic-cache@default-volatile-ds-group@volatileDsMemPlc> CAS
COMMIT name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7
thisNode=1c4499fb-5c33-43c4-8ad0-41c27c460e6c curTid=338
wroteOwner=1c4499fb-5c33-43c4-8ad0-41c27c460e6c wroteTid=338 wroteState=1
changed=true"
}
[2026-02-24T23:59:41.165Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,165",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=ba291a0e-4231-42c4-bb16-1e9d1d5cfd9b plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=1585993337
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.162Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,162",
"logLevel": "ERROR",
"threadName": "pub-#107%front-pod%",
"loggerName":
"org.apache.ignite.internal.processors.datastructures.GridCacheLockImpl",
"message":
"<ignite-sys-atomic-cache@default-volatile-ds-group@volatileDsMemPlc>
onNodeRemoved name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7
removed=d2648ea7-2a71-439d-856b-b102b50917f1 curOwner=null failoverSafe=true"
}
[2026-02-24T23:59:41.162Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,162",
"logLevel": "ERROR",
"threadName": "exchange-worker-#46%front-pod%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsSingleMessage
payload=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null,
partsSizes=null, partHistCntrs=null, err=null, client=true,
exchangeStartTime=1771977581161, finishMsg=null,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=36, minorTopVer=0],
discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=d2648ea7-2a71-439d-856b-b102b50917f1, consistentId=backend-pod-1,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 100.96.22.37, 127.0.0.1], sockAddrs=null,
discPort=47500, order=35, intOrder=20, loc=false,
ver=2.17.0#20250209-sha1:d53d4540, isClient=false], topVer=36,
msgTemplate=null,
span=org.apache.ignite.internal.processors.tracing.NoopSpan@1eaeb8ac,
nodeId8=1c4499fb, msg=Node left, type=NODE_LEFT, tstamp=1771977581153],
nodeId=d2648ea7, evt=NODE_LEFT], lastVer=GridCacheVersion [topVer=383456971,
order=1771977557071, nodeOrder=5, dataCenterId=0], super=GridCacheMessage
[msgId=13376, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion
[topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]"
}
[2026-02-24T23:59:41.160Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,160",
"logLevel": "WARN",
"threadName": "disco-event-worker-#45%front-pod%",
"loggerName":
"org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxPrepareFutureAdapter",
"message": "Received error when future is done [fut=MiniFuture [futId=1,
m=GridDistributedTxMapping [entries=LinkedHashSet [IgniteTxEntry
[txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=27,
val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
cacheId=1983463380], val=TxEntryValueHolder [val=UserCacheObjectImpl
[val=GridCacheLockState [cnt=1, threadId=338,
id=1c4499fb-5c33-43c4-8ad0-41c27c460e6c, gridStartTime=1771976936491,
failoverSafe=true, conditionMap=LinkedHashMap {}, signals=null, fair=false,
nodes=LinkedList [], changed=true], hasValBytes=true], op=UPDATE],
prevVal=TxEntryValueHolder [val=UserCacheObjectImpl [val=GridCacheLockState
[cnt=1, threadId=338, id=1c4499fb-5c33-43c4-8ad0-41c27c460e6c,
gridStartTime=1771976936491, failoverSafe=true, conditionMap=LinkedHashMap {},
signals=null, fair=false, nodes=LinkedList [], changed=true],
hasValBytes=true], op=UPDATE], oldVal=TxEntryValueHolder [val=null, op=NOOP],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [],
filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=27, val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=35, dataCenterId=0],
hash=-332917685, extras=null, flags=0]]], prepared=0, locked=true,
nodeId=d2648ea7-2a71-439d-856b-b102b50917f1, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0]]], explicitLock=false, dhtVer=null, last=false, nearEntries=0,
clientFirst=false, node=d2648ea7-2a71-439d-856b-b102b50917f1], done=true,
cancelled=false, err=null], err=class
o.a.i.i.cluster.ClusterTopologyCheckedException: Remote node left grid:
d2648ea7-2a71-439d-856b-b102b50917f1, tx=GridNearTxLocal
[mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false,
colocatedLocallyMapped=false, needCheckBackup=false, hasRemoteLocks=true,
trackTimeout=false, sysTime=0, sysStartTime=57416107986942,
prepareStartTime=57416107987111, prepareTime=24101206,
commitOrRollbackStartTime=57416132088586, commitOrRollbackTime=0, lb=null,
thread=qtp1855112877-338, mappings=IgniteTxMappingsImpl [],
super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=KeySetView
[], dhtNodes=KeySetView [], explicitLock=false, super=IgniteTxLocalAdapter
[doneFlag=0, completedBase=null, commitErr=null, depEnabled=false,
txState=IgniteTxStateImpl [activeCacheIds=[1983463380], recovery=false,
txMap=ArrayList [IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl
[part=27, val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
cacheId=1983463380], val=TxEntryValueHolder [val=UserCacheObjectImpl
[val=GridCacheLockState [cnt=1, threadId=338,
id=1c4499fb-5c33-43c4-8ad0-41c27c460e6c, gridStartTime=1771976936491,
failoverSafe=true, conditionMap=LinkedHashMap {}, signals=null, fair=false,
nodes=LinkedList [], changed=true], hasValBytes=true], op=UPDATE],
prevVal=TxEntryValueHolder [val=UserCacheObjectImpl [val=GridCacheLockState
[cnt=1, threadId=338, id=1c4499fb-5c33-43c4-8ad0-41c27c460e6c,
gridStartTime=1771976936491, failoverSafe=true, conditionMap=LinkedHashMap {},
signals=null, fair=false, nodes=LinkedList [], changed=true],
hasValBytes=true], op=UPDATE], oldVal=TxEntryValueHolder [val=null, op=NOOP],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [],
filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=27, val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=35, dataCenterId=0],
hash=-332917685, extras=null, flags=0]]], prepared=0, locked=true,
nodeId=d2648ea7-2a71-439d-856b-b102b50917f1, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0]]]], syncMode=null, super=IgniteTxAdapter
[xidVer=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0], writeVer=null, implicit=false, loc=true, threadId=338,
startTime=1771977581123, startTimeNanos=-1,
nodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c, needRetVal=false,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, internal=true, sys=true, plc=2, commitVer=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=5, dataCenterId=0],
finalizing=NONE, invalidParts=null, state=PREPARED, timedOut=false,
topVer=AffinityTopologyVersion [topVer=35, minorTopVer=1], subjId=null,
taskNameHash=0, storeEnabled=true, incSnpId=null, skipCompletedVers=false,
parentTx=null, duration=30ms, onePhaseCommit=true], size=1]]]]"
}
[2026-02-24T23:59:41.158Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,158",
"logLevel": "ERROR",
"threadName": "disco-event-worker-#45%front-pod%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=ba291a0e-4231-42c4-bb16-1e9d1d5cfd9b sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=1585993337
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishRequest
payload=GridDhtTxFinishRequest
[nearNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c, miniId=1,
super=GridDistributedTxFinishRequest [topVer=AffinityTopologyVersion
[topVer=35, minorTopVer=1],
futId=7c00f029c91-03f733e9-5c61-44fc-85fa-06c3a97c05b1, threadId=338,
commitVer=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0], invalidate=false, commit=true, baseVer=null, txSize=0,
sys=true, plc=2, taskNameHash=0, flags=2, syncMode=FULL_SYNC, txState=null,
super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=383456971,
order=1771977557071, nodeOrder=5, dataCenterId=0], committedVers=null,
rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0,
super=GridCacheMessage [msgId=13375, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=35, minorTopVer=1],
err=null, skipPrepare=false]]]]]"
}
[2026-02-24T23:59:41.136Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,136",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=ba291a0e-4231-42c4-bb16-1e9d1d5cfd9b plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=1585994263
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.135Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,134",
"logLevel": "ERROR",
"threadName": "sys-stripe-1-#2%backend-pod-1%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=ba291a0e-4231-42c4-bb16-1e9d1d5cfd9b sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=1585994263
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareRequest
payload=GridDhtTxPrepareRequest
[nearNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c,
futId=2c5c7129c91-37520a48-9fc5-4d81-a227-e2ff3af76afe, miniId=1,
topVer=AffinityTopologyVersion [topVer=35, minorTopVer=1],
invalidateNearEntries={}, nearWrites=null, owned=null,
nearXidVer=GridCacheVersion [topVer=383456971, order=1771977557071,
nodeOrder=5, dataCenterId=0], taskNameHash=0, preloadKeys=null,
skipCompletedVers=false, txLbl=null, super=GridDistributedTxPrepareRequest
[threadId=338, concurrency=PESSIMISTIC, isolation=REPEATABLE_READ,
writeVer=GridCacheVersion [topVer=383456971, order=1771977557076, nodeOrder=35,
dataCenterId=0], timeout=0, reads=null, writes=PredicateCollectionView
[IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=27,
val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
cacheId=1983463380], val=TxEntryValueHolder [val=CacheObjectImpl [val=null,
hasValBytes=true], op=UPDATE], prevVal=TxEntryValueHolder [val=null, op=NOOP],
oldVal=TxEntryValueHolder [val=null, op=NOOP], entryProcessorsCol=null, ttl=-1,
conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null,
filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false,
entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=27,
super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=27, val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=35, dataCenterId=0],
hash=-332917685, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc
[locs=LinkedList [GridCacheMvccCandidate
[nodeId=d2648ea7-2a71-439d-856b-b102b50917f1, ver=GridCacheVersion
[topVer=383456971, order=1771977557075, nodeOrder=35, dataCenterId=0],
threadId=338, id=137, topVer=AffinityTopologyVersion [topVer=35,
minorTopVer=1], reentry=null, otherNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c,
otherVer=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null,
serOrder=null, key=KeyCacheObjectImpl [part=27, val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=0,
locked=false, nodeId=null, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=383456971, order=1771977557075, nodeOrder=35,
dataCenterId=0]]], dhtVers=null, txSize=0, plc=2, txState=null,
flags=onePhase|last|sys, super=GridDistributedBaseMessage [ver=GridCacheVersion
[topVer=383456971, order=1771977557075, nodeOrder=35, dataCenterId=0],
committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage
[cacheId=0, super=GridCacheMessage [msgId=651, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=35, minorTopVer=1],
err=null, skipPrepare=false]]]]]"
}
[2026-02-24T23:59:41.133Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,133",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=1585993337
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.133Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,133",
"logLevel": "ERROR",
"threadName": "qtp1855112877-338",
"sessionID": "nil",
"correlationID": "nil",
"requestID": "nil",
"logCtxID": "84727db0-0bb5-4b93-a9df-f5ebbf1ff19c",
"tenantUser": "nil",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=d2648ea7-2a71-439d-856b-b102b50917f1 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=1585993337
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxPrepareRequest
payload=GridNearTxPrepareRequest
[futId=8c00f029c91-03f733e9-5c61-44fc-85fa-06c3a97c05b1, miniId=1,
topVer=AffinityTopologyVersion [topVer=35, minorTopVer=1], taskNameHash=0,
txLbl=null, flags=, super=GridDistributedTxPrepareRequest [threadId=338,
concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=5, dataCenterId=0],
timeout=0, reads=PredicateCollectionView [], writes=PredicateCollectionView
[IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=27,
val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
cacheId=1983463380], val=TxEntryValueHolder [val=UserCacheObjectImpl
[val=GridCacheLockState [cnt=1, threadId=338,
id=1c4499fb-5c33-43c4-8ad0-41c27c460e6c, gridStartTime=1771976936491,
failoverSafe=true, conditionMap=LinkedHashMap {}, signals=null, fair=false,
nodes=LinkedList [], changed=true], hasValBytes=true], op=UPDATE],
prevVal=TxEntryValueHolder [val=UserCacheObjectImpl [val=GridCacheLockState
[cnt=1, threadId=338, id=1c4499fb-5c33-43c4-8ad0-41c27c460e6c,
gridStartTime=1771976936491, failoverSafe=true, conditionMap=LinkedHashMap {},
signals=null, fair=false, nodes=LinkedList [], changed=true],
hasValBytes=true], op=UPDATE], oldVal=TxEntryValueHolder [val=null, op=NOOP],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [],
filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=27, val=GridCacheInternalKeyImpl
[name=CERTIFICATE_ROLES_MAP:82bdcce6-a165-4bfa-b9bf-56ee1ce9ede7,
grpName=default-volatile-ds-group@volatileDsMemPlc], hasValBytes=true],
val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=35, dataCenterId=0],
hash=-332917685, extras=null, flags=0]]], prepared=0, locked=true,
nodeId=d2648ea7-2a71-439d-856b-b102b50917f1, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0]]], dhtVers=null, txSize=0, plc=2, txState=null,
flags=onePhase|last|sys, super=GridDistributedBaseMessage [ver=GridCacheVersion
[topVer=383456971, order=1771977557071, nodeOrder=5, dataCenterId=0],
committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage
[cacheId=0, super=GridCacheMessage [msgId=13374, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=35, minorTopVer=1],
err=null, skipPrepare=false]]]]]"
}
[2026-02-24T23:59:41.132Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,132",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=d2648ea7-2a71-439d-856b-b102b50917f1 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.131Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,131",
"logLevel": "ERROR",
"threadName": "sys-stripe-3-#4%backend-pod-1%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearLockResponse
payload=GridNearLockResponse [pending=GridLeanSet [], miniId=1,
dhtVers=GridCacheVersion[] [GridCacheVersion [topVer=383456971,
order=1771977557071, nodeOrder=35, dataCenterId=0]],
mappedVers=GridCacheVersion[] [GridCacheVersion [topVer=383456971,
order=1771977557075, nodeOrder=35, dataCenterId=0]], clientRemapVer=null,
compatibleRemapVer=false, super=GridDistributedLockResponse
[futId=3c00f029c91-03f733e9-5c61-44fc-85fa-06c3a97c05b1, err=null,
vals=ArrayList [CacheObjectImpl [val=null, hasValBytes=true]],
super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=383456971,
order=1771977557071, nodeOrder=5, dataCenterId=0], committedVers=EmptyList [],
rolledbackVers=EmptyList [], cnt=1, super=GridCacheIdMessage
[cacheId=1983463380, super=GridCacheMessage [msgId=650, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0],
err=null, skipPrepare=false]]]]]"
}
[2026-02-24T23:59:41.129Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,129",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=27
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.128Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,128",
"logLevel": "ERROR",
"threadName": "qtp1855112877-338",
"sessionID": "nil",
"correlationID": "nil",
"requestID": "nil",
"logCtxID": "84727db0-0bb5-4b93-a9df-f5ebbf1ff19c",
"tenantUser": "nil",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=d2648ea7-2a71-439d-856b-b102b50917f1 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=27
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearLockRequest
payload=GridNearLockRequest [topVer=AffinityTopologyVersion [topVer=35,
minorTopVer=1], miniId=1, dhtVers=GridCacheVersion[] [null], taskNameHash=0,
createTtl=-1, accessTtl=-1, flags=7, txLbl=null,
super=GridDistributedLockRequest [nodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c,
nearXidVer=GridCacheVersion [topVer=383456971, order=1771977557071,
nodeOrder=5, dataCenterId=0], threadId=338,
futId=3c00f029c91-03f733e9-5c61-44fc-85fa-06c3a97c05b1, timeout=0, isInTx=true,
isInvalidate=false, isRead=true, isolation=REPEATABLE_READ, retVals=[true],
txSize=0, flags=0, keysCnt=1, super=GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=383456971, order=1771977557071, nodeOrder=5,
dataCenterId=0], committedVers=null, rolledbackVers=null, cnt=1,
super=GridCacheIdMessage [cacheId=1983463380, super=GridCacheMessage
[msgId=13373, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion
[topVer=35, minorTopVer=1], err=null, skipPrepare=false]]]]]"
}
[2026-02-24T23:59:41.127Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,127",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.126Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,125",
"logLevel": "ERROR",
"threadName": "sys-stripe-0-#1%backend-pod-1%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearSingleGetResponse
payload=GridNearSingleGetResponse [futId=1771976982719,
res=app.cfcs.CFCMetadata [idHash=1019494707, hash=-806951475, ... REDACTED
...], topVer=null, err=null, flags=0]"
}
[2026-02-24T23:59:41.126Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,126",
"logLevel": "ERROR",
"threadName": "qtp1855112877-338",
"sessionID": "nil",
"correlationID": "nil",
"requestID": "nil",
"logCtxID": "84727db0-0bb5-4b93-a9df-f5ebbf1ff19c",
"tenantUser": "nil",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=152
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearSingleGetRequest
payload=GridNearSingleGetRequest [futId=1771976982722,
key=app.cfcs.CFCIdentity [idHash=1739339080, hash=-79622183, ... REDACTED ...],
flags=1, topVer=AffinityTopologyVersion [topVer=35, minorTopVer=1],
taskNameHash=0, createTtl=-1, accessTtl=-1, txLbl=null]"
}
[2026-02-24T23:59:41.125Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,125",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
fromNodeId=d2648ea7-2a71-439d-856b-b102b50917f1 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.124Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,124",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=24
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.123Z - front-pod]
{
"timestamp": "2026-02-24 23:59:41,123",
"logLevel": "ERROR",
"threadName": "qtp1855112877-338",
"sessionID": "nil",
"correlationID": "nil",
"requestID": "nil",
"logCtxID": "84727db0-0bb5-4b93-a9df-f5ebbf1ff19c",
"tenantUser": "nil",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=1c4499fb-5c33-43c4-8ad0-41c27c460e6c
toNodeId=d2648ea7-2a71-439d-856b-b102b50917f1 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=24
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearSingleGetRequest
payload=GridNearSingleGetRequest [futId=1771976982719,
key=UserKeyCacheObjectImpl [part=24, val=mycontainer, hasValBytes=true],
flags=1, topVer=AffinityTopologyVersion [topVer=35, minorTopVer=1],
taskNameHash=0, createTtl=-1, accessTtl=-1, txLbl=null]"
}
[2026-02-24T23:59:41.120Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,120",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-2-#25%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=ba291a0e-4231-42c4-bb16-1e9d1d5cfd9b plc=3 ordered=false topicOrd=2
topic=TOPIC_TASK part=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.117Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,117",
"logLevel": "ERROR",
"threadName": "pool-29-thread-1",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=ba291a0e-4231-42c4-bb16-1e9d1d5cfd9b sameNode=false plc=3
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=0
topic=TOPIC_JOB ioTopicOrd=0 ioPart=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.GridJobExecuteRequest
payload=GridJobExecuteRequest
[sesId=0a5c7129c91-d2648ea7-2a71-439d-856b-b102b50917f1,
jobId=2a5c7129c91-d2648ea7-2a71-439d-856b-b102b50917f1,
startTaskTime=1771977581097, timeout=9223372036854775807,
taskName=org.apache.ignite.internal.CheckCpHistTask, userVer=0,
taskClsName=org.apache.ignite.internal.CheckCpHistTask, ldrParticipants=null,
cpSpi=null, createTime=1771977581117,
clsLdrId=5d1c7129c91-d2648ea7-2a71-439d-856b-b102b50917f1, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null]"
}
[2026-02-24T23:59:41.116Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,116",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=3 ordered=false topicOrd=2
topic=TOPIC_TASK part=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.113Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,113",
"logLevel": "ERROR",
"threadName": "pool-29-thread-1",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 sameNode=false plc=3
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=0
topic=TOPIC_JOB ioTopicOrd=0 ioPart=-2147483648
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.GridJobExecuteRequest
payload=GridJobExecuteRequest
[sesId=0a5c7129c91-d2648ea7-2a71-439d-856b-b102b50917f1,
jobId=1a5c7129c91-d2648ea7-2a71-439d-856b-b102b50917f1,
startTaskTime=1771977581097, timeout=9223372036854775807,
taskName=org.apache.ignite.internal.CheckCpHistTask, userVer=0,
taskClsName=org.apache.ignite.internal.CheckCpHistTask, ldrParticipants=null,
cpSpi=null, createTime=1771977581107,
clsLdrId=5d1c7129c91-d2648ea7-2a71-439d-856b-b102b50917f1, depMode=SHARED,
dynamicSiblings=false, forceLocDep=true, sesFullSup=false, internal=true,
topPred=null, part=-1, topVer=null, execName=null]"
}
[2026-02-24T23:59:41.109Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,109",
"logLevel": "ERROR",
"threadName": "sys-stripe-6-#7%backend-pod-1%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 sameNode=false plc=2
ordered=false async=false timeout=0 skipOnTimeout=false topicOrd=8
topic=TOPIC_CACHE ioTopicOrd=8 ioPart=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.distributed.near.GridNearSingleGetResponse
payload=GridNearSingleGetResponse [futId=1771976956740,
res=app.imdg.ignite.features.tasks.registry.ImdgScheduledTaskMetadata
[idHash=1440871452, hash=-443319403, lastRunStatus=null, errorMessage=null,
startedAt=1771976972752, initialDelay=30, finishedAt=null,
lastRunErrorMessage=null, result=null, isPeriodic=true,
task=app.imdg.ignite.features.tasks.executors.IgniteImdgScheduledExecutorService$EffectiveIgniteTask
[idHash=1719227076, hash=-842165524, isPeriodic=true,
task=java.util.concurrent.Executors$RunnableAdapter [idHash=1523220768,
hash=-1216062884, result=null, task=app.k8soperator.K8SOperator
[idHash=426525351, hash=1]], taskName=K8S_OPERATOR_TASK,
scheduledTaskRegistryId=K8S_OPERATOR_SCHEDULED_EXECUTOR], partitionKey=null,
errorClassName=null,
app.imdg.ignite.features.tasks.registry.ImdgScheduledTaskMetadata.submittedAt=1771976942585,
isAutoDisposable=false, executionCount=0, interval=300,
lastExecutionTimeMillis=0, taskName=K8S_OPERATOR_TASK, lastRunResult=null,
nextExecutionTimeMillis=1771976972585, lastRunErrorClassName=null,
app.imdg.ignite.features.tasks.registry.ImdgTaskMetadata.submittedAt=1771976942585,
timeUnit=SECONDS, status=RUNNING], topVer=null, err=null, flags=0]"
}
[2026-02-24T23:59:41.108Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,108",
"logLevel": "ERROR",
"threadName": "query-#64%backend-pod-1%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_SEND locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
toNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 sameNode=false plc=10
ordered=true async=false timeout=9223372036854775807 skipOnTimeout=false
topicOrd=-1 topic=T4 [topic=TOPIC_CACHE,
id1=56620808-72ee-3e1e-9cee-ec5750198283,
id2=1f73012b-df27-44e1-a8a6-9d9e1e101693, id3=12569] ioTopicOrd=-1 ioPart=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage
payloadCls=org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse
payload=GridCacheQueryResponse [finished=true, reqId=12569, err=null,
fields=false, metadata=null, idxQryMetadata=null]"
}
[2026-02-24T23:59:41.108Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,108",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=2 ordered=false topicOrd=8
topic=TOPIC_CACHE part=958
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.107Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,107",
"logLevel": "ERROR",
"threadName": "grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.internal.managers.communication.GridIoManager",
"message": "IO_DBG_RECV locNodeId=d2648ea7-2a71-439d-856b-b102b50917f1
fromNodeId=1f73012b-df27-44e1-a8a6-9d9e1e101693 plc=10 ordered=false topicOrd=8
topic=TOPIC_CACHE part=-1
ioMsgCls=org.apache.ignite.internal.managers.communication.GridIoMessage"
}
[2026-02-24T23:59:41.060Z - backend-pod-1]
{
"timestamp": "2026-02-24 23:59:41,053",
"logLevel": "INFO",
"threadName": "Thread-0",
"loggerName": "app.shutdown.ShutdownProcessController",
"message": "Calling Ignition.stop...",
"sgwLogType": "DEFAULT"
}
