Konstantin Orlov created IGNITE-19439:
-----------------------------------------
Summary: AssertionError in RemoveWriteOnGcInvokeClosure during
insertion
Key: IGNITE-19439
URL: https://issues.apache.org/jira/browse/IGNITE-19439
Project: Ignite
Issue Type: Bug
Reporter: Konstantin Orlov
Attachments: logs.txt
I run a test that in an endless loop inserts 10k rows , clears the table, and
starts over, but it failed on 4th iteration with AssertionError.
The test is as follow:
{code:java}
// org.apache.ignite.internal.sql.engine.ItDmlTest
private static final String STRING_PAYLOAD = IgniteTestUtils.randomString(new
Random(), 32);
@Test
public void test() throws SQLException {
int TABLE_SIZE = 10_000;
sql("CREATE TABLE t (id INT PRIMARY KEY, name VARCHAR, surname VARCHAR, age
TINYINT)");
int iteration = 1;
do {
System.out.println("Iteration " + (iteration++) + " started");
long before = System.currentTimeMillis();
for (int i = 1; i <= TABLE_SIZE; i++) {
sql("INSERT INTO t VALUES (?, ?, ?, ?)", i, STRING_PAYLOAD,
STRING_PAYLOAD, (byte) i);
if (i % 10 == 0) {
long after = System.currentTimeMillis();
System.out.println(i + " rows inserted. dT=" +
Duration.ofMillis(after - before));
before = after;
}
}
sql("DELETE FROM t");
} while (true);
} {code}
The assertion is:
{code:java}
023-05-08 20:17:51:016 +0300
[WARNING][%idt_n_0%JRaft-Request-Processor-0][ReplicaManager] Failed to process
replica request [request=ReadWriteMultiRowReplicaRequestImpl
[binaryRows=ArrayList [org.apache.ignite.internal.schema.row.Row@2b6ae605],
commitPartitionId=f6b9512f-b735-446e-85a3-d6b8636c847c_part_18,
groupId=f6b9512f-b735-446e-85a3-d6b8636c847c_part_1, requestType=RW_INSERT_ALL,
term=1, timestampLong=110334199137239040,
transactionId=0187fc5e-9a26-0000-0000-000063a26a19]]
java.util.concurrent.CompletionException:
org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
TraceId:c48307b5-59a0-4895-a79d-63153a9d47c6 Error removing row version from
version chain on garbage collection: [rowId=RowId [partitionId=1,
uuid=ca451c56-ea47-422e-b228-67c093ae1a10], rowTimestamp=HybridTimestamp
[time=110334012513058818], table=T, partitionId=1]
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
at
java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1113)
at
java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processMultiEntryAction$75(PartitionReplicaListener.java:1487)
at
java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
at
java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.processMultiEntryAction(PartitionReplicaListener.java:1445)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processRequest$2(PartitionReplicaListener.java:285)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.appendTxCommand(PartitionReplicaListener.java:1197)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.processRequest(PartitionReplicaListener.java:285)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$invoke$0(PartitionReplicaListener.java:274)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$39(RaftGroupServiceImpl.java:538)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.ignite.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:371)
at
org.apache.ignite.network.DefaultMessagingService.send0(DefaultMessagingService.java:194)
at
org.apache.ignite.network.DefaultMessagingService.respond(DefaultMessagingService.java:137)
at
org.apache.ignite.network.MessagingService.respond(MessagingService.java:89)
at
org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$NetworkRpcContext.sendResponse(IgniteRpcServer.java:233)
at
org.apache.ignite.raft.jraft.rpc.RpcRequestProcessor.handleRequest(RpcRequestProcessor.java:52)
at
org.apache.ignite.raft.jraft.rpc.RpcRequestProcessor.handleRequest(RpcRequestProcessor.java:29)
at
org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:192)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
TraceId:c48307b5-59a0-4895-a79d-63153a9d47c6 Error removing row version from
version chain on garbage collection: [rowId=RowId [partitionId=1,
uuid=ca451c56-ea47-422e-b228-67c093ae1a10], rowTimestamp=HybridTimestamp
[time=110334012513058818], table=T, partitionId=1]
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.removeWriteOnGc(AbstractPageMemoryMvPartitionStorage.java:968)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.vacuum(AbstractPageMemoryMvPartitionStorage.java:952)
at
org.apache.ignite.internal.storage.MvPartitionStorage.lambda$pollForVacuum$0(MvPartitionStorage.java:292)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:143)
at
org.apache.ignite.internal.storage.MvPartitionStorage.pollForVacuum(MvPartitionStorage.java:282)
at
org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.pollForVacuum(SnapshotAwarePartitionDataStorage.java:141)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.internalVacuum(StorageUpdateHandler.java:345)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$vacuumBatch$4(StorageUpdateHandler.java:332)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:155)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:774)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:145)
at
org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:66)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.vacuumBatch(StorageUpdateHandler.java:332)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.executeBatchGc(StorageUpdateHandler.java:202)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdateAll(StorageUpdateHandler.java:192)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.applyUpdateAllCommand(PartitionReplicaListener.java:1600)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processMultiEntryAction$72(PartitionReplicaListener.java:1487)
at
java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
... 28 more
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
IGN-CMN-65535 TraceId:0f815e82-c0df-4f6e-8784-be38e9ed94cc B+Tree is corrupted
[groupId=1, pageIds=[562954248388665], groupName=T, msg=Runtime failure on
search row:
org.apache.ignite.internal.storage.pagememory.mv.VersionChainKey@13e0d6a6]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6724)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2135)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.removeWriteOnGc(AbstractPageMemoryMvPartitionStorage.java:961)
... 45 more
Caused by: java.lang.AssertionError: rowId=RowId [partitionId=1,
uuid=ca451c56-ea47-422e-b228-67c093ae1a10], storage=table=T, partitionId=1
at
org.apache.ignite.internal.storage.pagememory.mv.RemoveWriteOnGcInvokeClosure.call(RemoveWriteOnGcInvokeClosure.java:70)
at
org.apache.ignite.internal.storage.pagememory.mv.RemoveWriteOnGcInvokeClosure.call(RemoveWriteOnGcInvokeClosure.java:42)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.invokeClosure(BplusTree.java:4298)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2209)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2191)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
... 46 more
{code}
Full log is attached below. [^logs.txt]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
