[
https://issues.apache.org/jira/browse/IGNITE-23803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-23803:
---------------------------------------
Description:
Node leave caused the following:
2024-11-27 15:56:57:316 +0000
[WARNING][%ReproducerTest_cluster_2%connection-maintenance-0][CatalogManagerImpl]
Failed to apply catalog update.
java.util.concurrent.CompletionException:
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.handleThrowable(RaftGroupServiceImpl.java:641)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$49(RaftGroupServiceImpl.java:618)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.ignite.internal.network.OutNetworkObject.failAcknowledgement(OutNetworkObject.java:95)
at
org.apache.ignite.internal.network.recovery.RecoveryDescriptor.dispose(RecoveryDescriptor.java:265)
at
org.apache.ignite.internal.network.netty.ConnectionManager.blockAndDisposeDescriptor(ConnectionManager.java:679)
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:647)
at
org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:623)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.ignite.internal.network.RecipientLeftException:
IGN-NETWORK-5 TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
... 7 more
2024-11-27 15:56:57:317 +0000
[INFO][%ReproducerTest_cluster_2%connection-maintenance-0][WatchProcessor]
Notification chain encountered an error, so no notifications will be ever fired
for subsequent revisions until a restart. Notifying the FailureManager
2024-11-27 15:56:57:318 +0000
[ERROR][%ReproducerTest_cluster_2%connection-maintenance-0][FailureManager]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR]
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
at
org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:623)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2024-11-27 15:56:57:318 +0000
[WARNING][%ReproducerTest_cluster_2%connection-maintenance-0][UpdateLogImpl]
Unable to process catalog event
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
at
org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:623)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
It seems that, as a reaction to some Catalog event application, a Catalog
listener tried to send a message to another node, the recipient node left the
topology, so the send failed with RecipientLeftException, and the failure
popped to the CatalogManager which considers any failure of any of its
listeners as a fatal error and notifies the FailureHandler. Even worse, as the
same exception then reached WatchProcessor, it halted subsequent Metastorage
events processing:
2024-11-27 15:56:57:317 +0000
[INFO][%Gg41011ReproducerTest_cluster_2%connection-maintenance-0][WatchProcessor]
Notification chain encountered an error, so no notifications will be ever
fired for subsequent revisions until a restart. Notifying the FailureManager
2024-11-27 15:56:57:318 +0000
[ERROR][%Gg41011ReproducerTest_cluster_2%connection-maintenance-0][FailureManager]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR]
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
> Node leave causes a critical failure on schema update application
> -----------------------------------------------------------------
>
> Key: IGNITE-23803
> URL: https://issues.apache.org/jira/browse/IGNITE-23803
> Project: Ignite
> Issue Type: Bug
> Reporter: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
>
> Node leave caused the following:
>
> 2024-11-27 15:56:57:316 +0000
> [WARNING][%ReproducerTest_cluster_2%connection-maintenance-0][CatalogManagerImpl]
> Failed to apply catalog update.
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
> TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.handleThrowable(RaftGroupServiceImpl.java:641)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$49(RaftGroupServiceImpl.java:618)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.ignite.internal.network.OutNetworkObject.failAcknowledgement(OutNetworkObject.java:95)
> at
> org.apache.ignite.internal.network.recovery.RecoveryDescriptor.dispose(RecoveryDescriptor.java:265)
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.blockAndDisposeDescriptor(ConnectionManager.java:679)
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:647)
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:623)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.ignite.internal.network.RecipientLeftException:
> IGN-NETWORK-5 TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
> ... 7 more
> 2024-11-27 15:56:57:317 +0000
> [INFO][%ReproducerTest_cluster_2%connection-maintenance-0][WatchProcessor]
> Notification chain encountered an error, so no notifications will be ever
> fired for subsequent revisions until a restart. Notifying the FailureManager
> 2024-11-27 15:56:57:318 +0000
> [ERROR][%ReproducerTest_cluster_2%connection-maintenance-0][FailureManager]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR]
> org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
> TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:623)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2024-11-27 15:56:57:318 +0000
> [WARNING][%ReproducerTest_cluster_2%connection-maintenance-0][UpdateLogImpl]
> Unable to process catalog event
> org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
> TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:623)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> It seems that, as a reaction to some Catalog event application, a Catalog
> listener tried to send a message to another node, the recipient node left the
> topology, so the send failed with RecipientLeftException, and the failure
> popped to the CatalogManager which considers any failure of any of its
> listeners as a fatal error and notifies the FailureHandler. Even worse, as
> the same exception then reached WatchProcessor, it halted subsequent
> Metastorage events processing:
>
> 2024-11-27 15:56:57:317 +0000
> [INFO][%Gg41011ReproducerTest_cluster_2%connection-maintenance-0][WatchProcessor]
> Notification chain encountered an error, so no notifications will be ever
> fired for subsequent revisions until a restart. Notifying the FailureManager
> 2024-11-27 15:56:57:318 +0000
> [ERROR][%Gg41011ReproducerTest_cluster_2%connection-maintenance-0][FailureManager]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR]
> org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
> TraceId:61b649e8-3f0b-48b7-9bc1-e2f5a9060cbb
> at
> org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:644)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
