Assen Sharlandjiev created IGNITE-19146:
-------------------------------------------
Summary: java.lang.ArrayIndexOutOfBoundsException in
IgniteUtils.java:10751
Key: IGNITE-19146
URL: https://issues.apache.org/jira/browse/IGNITE-19146
Project: Ignite
Issue Type: Bug
Components: general
Affects Versions: 2.14
Reporter: Assen Sharlandjiev
Hi, I'm getting the following exception when the cluster is under load.
I'm also able to reproduce this error consistently.
{noformat}
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | 2023-03-28T13:31:05.482Z
ERROR 1 --- [mmunicationSpi%] o.a.i.i.m.communication.GridIoManager : Failed
to process message (will ignore): GridIoMessage [plc=0, topic=null,
topicOrd=-1, ordered=false, timeout=-1, skipOnTimeout=true,
msg=GridCacheQueryRequest [id=17623, cacheName=com.azdio.slate.domain.Slate,
type=SCAN, fields=false, clause=null, idxQryDesc=null, limit=0, clsName=null,
keyValFilter=null, rdc=null, trans=null, pageSize=1024, incBackups=false,
cancel=false, incMeta=false, all=false, keepBinary=true, taskHash=0, part=-1,
topVer=AffinityTopologyVersion [topVer=481, minorTopVer=0], mvccSnapshot=null,
flags=0, super=GridCacheIdMessage [cacheId=1617697070, super=GridCacheMessage
[msgId=17624, depInfo=GridDeploymentInfoBean
[clsLdrId=17177682781-a1fbd831-473b-4c94-bf39-5dfcd4f22cb3, depMode=SHARED,
userVer=0, locDepOwner=false, participants=null],
lastAffChangedTopVer=AffinityTopologyVersion [topVer=401, minorTopVer=1],
err=null, skipPrepare=false]]]]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 |
ignite_ignite.3.q1dazbnptk98@digicel-int-121 |
org.apache.ignite.IgniteCheckedException: null
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10751)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.managers.communication.GridIoManager.onMessage0(GridIoManager.java:1273)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.managers.communication.GridIoManager.access$300(GridIoManager.java:243)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.managers.communication.GridIoManager$2.onMessage(GridIoManager.java:509)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.notifyListener(TcpCommunicationSpi.java:1220)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$1.onMessage(TcpCommunicationSpi.java:689)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$1.onMessage(TcpCommunicationSpi.java:687)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.spi.communication.tcp.internal.InboundConnectionHandler.onMessage(InboundConnectionHandler.java:393)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.spi.communication.tcp.internal.InboundConnectionHandler.onMessage(InboundConnectionHandler.java:79)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:133)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3734)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | Caused by:
java.lang.ArrayIndexOutOfBoundsException: null
{noformat}
after this one, i see a lot of:
{noformat}
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | 2023-03-28T13:31:06.483Z
ERROR 1 --- [mmunicationSpi%] o.a.i.s.c.tcp.TcpCommunicationSpi : Failed
to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=13,
bytesRcvd=234716720, bytesSent=211278209, bytesRcvd0=1724539,
bytesSent0=1710663, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-13, igniteInstanceName=TcpCommunicationSpi,
finished=false, heartbeatTs=1680010265474, hashCode=1814229823,
interrupted=false,
runner=grid-nio-worker-tcp-comm-13-#100%TcpCommunicationSpi%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=9 lim=13980 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=8800, resendCnt=0, rcvCnt=8999,
sentCnt=8801, reserved=true, lastAck=8992, nodeLeft=false,
node=ZookeeperClusterNode [id=a1fbd831-473b-4c94-bf39-5dfcd4f22cb3,
addrs=[10.0.1.130], order=481, loc=false, client=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=8800, resendCnt=0, rcvCnt=8999,
sentCnt=8801, reserved=true, lastAck=8992, nodeLeft=false,
node=ZookeeperClusterNode [id=a1fbd831-473b-4c94-bf39-5dfcd4f22cb3,
addrs=[10.0.1.130], order=481, loc=false, client=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
closeSocket=true,
outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric@69a257d1,
super=GridNioSessionImpl [locAddr=/10.0.1.67:47100, rmtAddr=/10.0.1.130:48418,
createTime=1680010081358, closeTime=0, bytesSent=7302922, bytesRcvd=5919769,
bytesSent0=1710663, bytesRcvd0=1724539, sndSchedTime=1680010081358,
lastSndTime=1680010265474, lastRcvTime=1680010265474, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@6818fc16, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 |
ignite_ignite.3.q1dazbnptk98@digicel-int-121 |
org.apache.ignite.IgniteException: Invalid message type: 2057
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.managers.communication.IgniteMessageFactoryImpl.create(IgniteMessageFactoryImpl.java:133)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper$2.create(GridNioServerWrapper.java:813)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridDirectParser.decode(GridDirectParser.java:81)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:113)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:133)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3734)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
~[ignite-core-2.14.0.jar:2.14.0]
ignite_ignite.3.q1dazbnptk98@digicel-int-121 | at
java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
{noformat}
and on the client's side I see:
{noformat}
slate_slate-api.1.tnhgicxawhbr@digicel-int-122 | 2023-03-28T13:31:06.486Z
INFO 1 --- [mmunicationSpi%] o.a.i.s.c.tcp.TcpCommunicationSpi :
Accepted incoming communication connection [locAddr=/10.0.1.130:47100,
rmtAddr=/10.0.1.67:59680]
slate_slate-api.1.tnhgicxawhbr@digicel-int-122 | 2023-03-28T13:31:07.880Z
INFO 1 --- [mmunicationSpi%] o.a.i.s.c.tcp.TcpCommunicationSpi :
Accepted incoming communication connection [locAddr=/10.0.1.130:47100,
rmtAddr=/10.0.1.71:45326]
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
