[
https://issues.apache.org/jira/browse/IGNITE-27125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18042236#comment-18042236
]
Vladislav Pyatkov commented on IGNITE-27125:
--------------------------------------------
In the other failed test
(ItHighAvailablePartitionsRecoveryNoStaleRecoveriesTest#testHaRecoveryWhenMajorityLossNoRedundantResetR
equests), I noticed that the exception happens during topology events
processing. The exception looks like MG sends two (probably simular) updates
with the same revision (violating a guarantee of unique update notification for
MG watch)
{noformat}
[05:25:21] : [:ignite-table:integrationTest]
[2025-11-29T05:25:21,485][INFO ][Test
worker][ItHighAvailablePartitionsRecoveryNoStaleRecoveriesTest] >>> Starting
test:
ItHighAvailablePartitionsRecoveryNoStaleRecoveriesTest#testHaRecoveryWhenMajorityLossNoRedundantResetR
equests, displayName: testHaRecoveryWhenMajorityLossNoRedundantResetRequests()
[05:25:23]W: [:ignite-table:integrationTest]
[2025-11-29T05:25:23,073][ERROR][%ihaprnsrt_thrwmlnrrr_3347%metastorage-watch-executor-0][FailureManager]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=Abst
ractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR,
failureCtxId=0f1757a6-7170-4b62-ad8f-dfd5b3300d00]
[05:25:23]W: [:ignite-table:integrationTest]
[05:25:23]W: [:ignite-table:integrationTest]
org.apache.ignite.internal.failure.StackTraceCapturingException: Unknown error
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:184)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:161)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyFailureHandlerOnFirstFailureInNotificationChain(WatchProcessor.java:434)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$3(WatchProcessor.java:234)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1163)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.lang.Thread.run(Thread.java:833)
[05:25:23]W: [:ignite-table:integrationTest] Caused by:
java.util.concurrent.CompletionException: java.lang.AssertionError: Causality
token must be greater than the last completed: [token=13, lastCompleted=13]
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152)
[05:25:23]W: [:ignite-table:integrationTest] ... 4 more
[05:25:23]W: [:ignite-table:integrationTest] Caused by:
java.lang.AssertionError: Causality token must be greater than the last
completed: [token=13, lastCompleted=13]
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:276)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:107)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:477)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatchesInternal$7(WatchProcessor.java:259)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:933)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.metastorage.server.WatchProcessor.inBusyLockAsync(WatchProcessor.java:185)
[05:25:23]W: [:ignite-table:integrationTest] at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$2(WatchProcessor.java:231)
[05:25:23]W: [:ignite-table:integrationTest] at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
[05:25:23]W: [:ignite-table:integrationTest] ... 4 more
{noformat}
It can be a crucial thing in a way to reproduce this.
[^_Integration_Tests_Integration_Table_20340.log.zip]
> ItIgniteStopTest#testNodesCouldBeStoppedEvenIfMetastorageIsUnavailable failed
> with Causality token must be greater than the last complete
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-27125
> URL: https://issues.apache.org/jira/browse/IGNITE-27125
> Project: Ignite
> Issue Type: Bug
> Reporter: Mikhail Efremov
> Priority: Major
> Labels: MakeTeamcityGreenAgain, ignite-3
> Attachments: _Integration_Tests_Integration_Runner_437.log.zip,
> _Integration_Tests_Integration_Table_20340.log.zip
>
>
> *Subject test*
> {{ItIgniteStopTest#testNodesCouldBeStoppedEvenIfMetastorageIsUnavailable}}
> *Reason to fail*
> {code:java}
> [2025-11-20T06:48:28,273][WARN
> ][%iist_tncbseimiu_3346%MessagingService-inbound-Default-0-0][RaftGroupServiceImpl]
> All peers are unavailable, going to keep retrying until timeout [peers =
> [Peer [consistentId=iist_tncbseimiu_3344, idx=0]], group = metastorage_group,
> trace ID: d1fcf253-3c0f-4f1a-bcfa-2c13564adafd, request
> org.apache.ignite.raft.jraft.rpc.GetLeaderRequestImpl, origin command
> org.apache.ignite.internal.metastorage.command.GetCurrentRevisionsCommandImpl,
> instance=org.apache.ignite.internal.raft.RaftGroupServiceImpl@49681063].
> [2025-11-20T06:48:28,514][ERROR][%iist_tncbseimiu_3345%metastorage-watch-executor-0][WatchProcessor]
> Notification chain encountered an error, so no notifications will be ever
> fired for subsequent revisions until a restart. Notifying the FailureManager.
> Additional info: 'Keys of updated entries:
> catalog.update.1,catalog.version,icp.4b68f1f3-1268-47dd-926c-72f1eecb1f4a_cnt_2'
> [2025-11-20T06:48:28,517][ERROR][%iist_tncbseimiu_3345%metastorage-watch-executor-0][FailureManager]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR,
> failureCtxId=7392760b-6d4c-4afe-a70b-9092f2cbee09]
> org.apache.ignite.internal.failure.StackTraceCapturingException: Unknown error
> at
> org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:192)
> at
> org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:169)
> at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyFailureHandlerOnFirstFailureInNotificationChain(WatchProcessor.java:442)
> at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$3(WatchProcessor.java:242)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1085)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.util.concurrent.CompletionException:
> java.lang.AssertionError: Causality token must be greater than the last
> completed: [token=5, lastCompleted=5]
> at
> java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
> at
> java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1074)
> ... 4 more
> Caused by: java.lang.AssertionError: Causality token must be greater than the
> last completed: [token=5, lastCompleted=5]
> at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:284)
> at
> org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:115)
> at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:485)
> at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatchesInternal$7(WatchProcessor.java:267)
> at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:941)
> at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.inBusyLockAsync(WatchProcessor.java:193)
> at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$2(WatchProcessor.java:239)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
> ... 4 more
> {code}
> *Failed TC build*
> [TC
> link|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3x_Test_RunAllTests/9670351]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
