[
https://issues.apache.org/jira/browse/IGNITE-24185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17912187#comment-17912187
]
Kirill Tkalenko commented on IGNITE-24185:
------------------------------------------
According to the attached logs, a catalog compaction occurred on the running
cluster, which means that if there is an attempt to read from the catalog on a
catalog version (or timestamp) less than or equal to the existing one, we will
receive errors, which is what happened.
Message about successful directory compaction from the logs:
{noformat}
2025-01-08 10:26:49:908 +0000
[INFO][%node1%low-watermark-updater-0][CatalogCompactionRunner] Catalog
compaction started [lowWaterMark=HybridTimestamp [physical=2025-01-08
10:16:49:407 +0000, logical=0, composite=113792215246897152]].
2025-01-08 10:26:49:912 +0000
[INFO][%node1%low-watermark-updater-0][LowWatermarkImpl] Successful low
watermark update: HybridTimestamp [physical=2025-01-08 10:16:49:407 +0000,
logical=0, composite=113792215246897152]
2025-01-08 10:26:50:102 +0000
[INFO][%node1%JRaft-FSMCaller-Disruptormetastorage_stripe_0-0][CatalogCompactionRunner]
Catalog compaction completed successfully [timestamp=113792215246700544].
2025-01-08 10:26:50:103 +0000
[INFO][%node1%metastorage-watch-executor-2][CatalogManagerImpl] Catalog history
was truncated up to version=14
{noformat}
After this, after some time, the cluster is restarted, and the nodes cannot
start, since they are trying to read from the catalog according to the time of
the metastorage revision in which the system properties were last updated, and
this coincides with the time of the very first cluster launch.
We need to fix this.
> Fix flaky
> ItMetaStorageCompactionTriggerTest#testMetastorageCompactionOccursOnAllNodes
> --------------------------------------------------------------------------------------
>
> Key: IGNITE-24185
> URL: https://issues.apache.org/jira/browse/IGNITE-24185
> Project: Ignite
> Issue Type: Bug
> Reporter: Kirill Tkalenko
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Need to fix flaky
> *org.apache.ignite.internal.metastorage.impl.ItMetaStorageCompactionTriggerTest#testMetastorageCompactionOccursOnAllNodes*.
> [TC|https://ci.ignite.apache.org/test/-4127619255405515877?currentProjectId=ApacheIgnite3xGradle_Test_IntegrationTests]
> link.
> {noformat}
> org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
> at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
> at
> app//org.apache.ignite.internal.metastorage.impl.ItMetaStorageCompactionTriggerTest.testMetastorageCompactionOccursOnAllNodes(ItMetaStorageCompactionTriggerTest.java:95)
> at java.base@17.0.6/java.lang.reflect.Method.invoke(Method.java:568)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
> at
> java.base@17.0.6/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> at
> java.base@17.0.6/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
> at
> java.base@17.0.6/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.base@17.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at
> java.base@17.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.base@17.0.6/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
> at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
