[
https://issues.apache.org/jira/browse/IGNITE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Orlov updated IGNITE-19439:
--------------------------------------
Ignite Flags: (was: Docs Required,Release Notes Required)
> AssertionError in RemoveWriteOnGcInvokeClosure during insertion
> ---------------------------------------------------------------
>
> Key: IGNITE-19439
> URL: https://issues.apache.org/jira/browse/IGNITE-19439
> Project: Ignite
> Issue Type: Bug
> Reporter: Konstantin Orlov
> Priority: Major
> Labels: ignite-3
> Attachments: logs.txt
>
>
> I run a test that in an endless loop inserts 10k rows , clears the table, and
> starts over, but it failed on 4th iteration with AssertionError.
> The test is as follow:
> {code:java}
> // org.apache.ignite.internal.sql.engine.ItDmlTest
> private static final String STRING_PAYLOAD = IgniteTestUtils.randomString(new
> Random(), 32);
> @Test
> public void test() throws SQLException {
> int TABLE_SIZE = 10_000;
> sql("CREATE TABLE t (id INT PRIMARY KEY, name VARCHAR, surname VARCHAR,
> age TINYINT)");
> int iteration = 1;
> do {
> System.out.println("Iteration " + (iteration++) + " started");
> long before = System.currentTimeMillis();
> for (int i = 1; i <= TABLE_SIZE; i++) {
> sql("INSERT INTO t VALUES (?, ?, ?, ?)", i, STRING_PAYLOAD,
> STRING_PAYLOAD, (byte) i);
> if (i % 10 == 0) {
> long after = System.currentTimeMillis();
> System.out.println(i + " rows inserted. dT=" +
> Duration.ofMillis(after - before));
> before = after;
> }
> }
> sql("DELETE FROM t");
> } while (true);
> } {code}
> The assertion is:
> {code:java}
> 023-05-08 20:17:51:016 +0300
> [WARNING][%idt_n_0%JRaft-Request-Processor-0][ReplicaManager] Failed to
> process replica request [request=ReadWriteMultiRowReplicaRequestImpl
> [binaryRows=ArrayList [org.apache.ignite.internal.schema.row.Row@2b6ae605],
> commitPartitionId=f6b9512f-b735-446e-85a3-d6b8636c847c_part_18,
> groupId=f6b9512f-b735-446e-85a3-d6b8636c847c_part_1,
> requestType=RW_INSERT_ALL, term=1, timestampLong=110334199137239040,
> transactionId=0187fc5e-9a26-0000-0000-000063a26a19]]
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:c48307b5-59a0-4895-a79d-63153a9d47c6 Error removing row version from
> version chain on garbage collection: [rowId=RowId [partitionId=1,
> uuid=ca451c56-ea47-422e-b228-67c093ae1a10], rowTimestamp=HybridTimestamp
> [time=110334012513058818], table=T, partitionId=1]
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1113)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processMultiEntryAction$75(PartitionReplicaListener.java:1487)
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.processMultiEntryAction(PartitionReplicaListener.java:1445)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processRequest$2(PartitionReplicaListener.java:285)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.appendTxCommand(PartitionReplicaListener.java:1197)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.processRequest(PartitionReplicaListener.java:285)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$invoke$0(PartitionReplicaListener.java:274)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$39(RaftGroupServiceImpl.java:538)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.ignite.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:371)
> at
> org.apache.ignite.network.DefaultMessagingService.send0(DefaultMessagingService.java:194)
> at
> org.apache.ignite.network.DefaultMessagingService.respond(DefaultMessagingService.java:137)
> at
> org.apache.ignite.network.MessagingService.respond(MessagingService.java:89)
> at
> org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$NetworkRpcContext.sendResponse(IgniteRpcServer.java:233)
> at
> org.apache.ignite.raft.jraft.rpc.RpcRequestProcessor.handleRequest(RpcRequestProcessor.java:52)
> at
> org.apache.ignite.raft.jraft.rpc.RpcRequestProcessor.handleRequest(RpcRequestProcessor.java:29)
> at
> org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:192)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:c48307b5-59a0-4895-a79d-63153a9d47c6 Error removing row version from
> version chain on garbage collection: [rowId=RowId [partitionId=1,
> uuid=ca451c56-ea47-422e-b228-67c093ae1a10], rowTimestamp=HybridTimestamp
> [time=110334012513058818], table=T, partitionId=1]
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.removeWriteOnGc(AbstractPageMemoryMvPartitionStorage.java:968)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.vacuum(AbstractPageMemoryMvPartitionStorage.java:952)
> at
> org.apache.ignite.internal.storage.MvPartitionStorage.lambda$pollForVacuum$0(MvPartitionStorage.java:292)
> at
> org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:143)
> at
> org.apache.ignite.internal.storage.MvPartitionStorage.pollForVacuum(MvPartitionStorage.java:282)
> at
> org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.pollForVacuum(SnapshotAwarePartitionDataStorage.java:141)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.internalVacuum(StorageUpdateHandler.java:345)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$vacuumBatch$4(StorageUpdateHandler.java:332)
> at
> org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:155)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:774)
> at
> org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:145)
> at
> org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:66)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.vacuumBatch(StorageUpdateHandler.java:332)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.executeBatchGc(StorageUpdateHandler.java:202)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdateAll(StorageUpdateHandler.java:192)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.applyUpdateAllCommand(PartitionReplicaListener.java:1600)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processMultiEntryAction$72(PartitionReplicaListener.java:1487)
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> ... 28 more
> Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
> IGN-CMN-65535 TraceId:0f815e82-c0df-4f6e-8784-be38e9ed94cc B+Tree is
> corrupted [groupId=1, pageIds=[562954248388665], groupName=T, msg=Runtime
> failure on search row:
> org.apache.ignite.internal.storage.pagememory.mv.VersionChainKey@13e0d6a6]
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6724)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2135)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.removeWriteOnGc(AbstractPageMemoryMvPartitionStorage.java:961)
> ... 45 more
> Caused by: java.lang.AssertionError: rowId=RowId [partitionId=1,
> uuid=ca451c56-ea47-422e-b228-67c093ae1a10], storage=table=T, partitionId=1
> at
> org.apache.ignite.internal.storage.pagememory.mv.RemoveWriteOnGcInvokeClosure.call(RemoveWriteOnGcInvokeClosure.java:70)
> at
> org.apache.ignite.internal.storage.pagememory.mv.RemoveWriteOnGcInvokeClosure.call(RemoveWriteOnGcInvokeClosure.java:42)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.invokeClosure(BplusTree.java:4298)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2209)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2191)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
> ... 46 more
> {code}
> Full log is attached below. [^logs.txt]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
