[
https://issues.apache.org/jira/browse/IGNITE-25267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-25267:
---------------------------------------
Attachment: _Integration_Tests_Module_CLI_36404.log.zip
> Handle closed storages in IndexBuildController
> ----------------------------------------------
>
> Key: IGNITE-25267
> URL: https://issues.apache.org/jira/browse/IGNITE-25267
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
> Attachments: _Integration_Tests_Module_CLI_36404.log.zip
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a node initiates stop while an index is being built, IndexBuildController
> might encounter StorageClosedException when initiating index build. Such
> exception should just be ignored instead of being reported to
> FailureProcessor. We can ignore it as after node start the index build will
> be reattempted (if the node becomes a primary again).
> [https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_IntegrationTests_ModuleCli/9092932]
>
> [13:02:51]W:
> [2025-04-29T09:02:51,293][ERROR][%irpct_n_3345%metastorage-watch-executor-3][FailureManager]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR]
> [13:02:51]W: org.apache.ignite.internal.failure.StackTraceCapturingException:
> Unknown error
> [13:02:51]W: at
> org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:161)
> ~[ignite-failure-handler-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:138)
> ~[ignite-failure-handler-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.lambda$onPrimaryReplicaElected$11(IndexBuildController.java:339)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.util.CompletableFutures.lambda$copyStateTo$2(CompletableFutures.java:160)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.copyState(BaseVersionedValue.java:335)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.complete(BaseVersionedValue.java:219)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.lambda$completeInternal$2(IncrementalVersionedValue.java:283)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:283)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:112)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.lambda$dependingOn$0(IncrementalVersionedValue.java:83)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.lambda$notifyCompletionListeners$6(BaseVersionedValue.java:367)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.notifyCompletionListeners(BaseVersionedValue.java:362)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.complete(BaseVersionedValue.java:228)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.lambda$completeInternal$2(IncrementalVersionedValue.java:283)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:283)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:112)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.lambda$dependingOn$0(IncrementalVersionedValue.java:83)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.lambda$notifyCompletionListeners$6(BaseVersionedValue.java:367)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.notifyCompletionListeners(BaseVersionedValue.java:362)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.BaseVersionedValue.complete(BaseVersionedValue.java:228)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.lambda$completeInternal$2(IncrementalVersionedValue.java:283)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:283)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:112)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:446)
> ~[ignite-metastorage-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatchesInternal$7(WatchProcessor.java:262)
> ~[ignite-metastorage-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:926)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.inBusyLockAsync(WatchProcessor.java:186)
> ~[ignite-metastorage-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$2(WatchProcessor.java:230)
> ~[ignite-metastorage-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> [?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> [?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> [13:02:51]W: at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
> [13:02:51]W: Caused by: java.util.concurrent.CompletionException:
> org.apache.ignite.internal.storage.StorageClosedException: IGN-CMN-65535
> TraceId:4a982a7f-fa01-43db-9637-555fc8b9ee2e Storage is already closed:
> [tableId=19]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:761)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:735)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2182)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.lambda$onPrimaryReplicaElected$9(IndexBuildController.java:320)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> ~[?:?]
> [13:02:51]W: ... 50 more
> [13:02:51]W: Caused by:
> org.apache.ignite.internal.storage.StorageClosedException: Storage is already
> closed: [tableId=19]
> [13:02:51]W: at
> org.apache.ignite.internal.storage.util.StorageUtils.throwExceptionDependingOnStorageState(StorageUtils.java:140)
> ~[ignite-storage-api-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.busy(AbstractPageMemoryTableStorage.java:267)
> ~[ignite-storage-page-memory-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.getIndex(AbstractPageMemoryTableStorage.java:380)
> ~[ignite-storage-page-memory-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.storage.engine.ThreadAssertingMvTableStorage.getIndex(ThreadAssertingMvTableStorage.java:136)
> ~[ignite-storage-api-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.indexStorage(IndexBuildController.java:557)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.scheduleBuildIndexAfterDisasterRecovery(IndexBuildController.java:511)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.lambda$tryScheduleBuildIndexesForNewPrimaryReplica$12(IndexBuildController.java:375)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:905)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.tryScheduleBuildIndexesForNewPrimaryReplica(IndexBuildController.java:353)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.lambda$onPrimaryReplicaElected$8(IndexBuildController.java:320)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:757)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:735)
> ~[?:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2182)
> ~[?:?]
> [13:02:51]W: at
> org.apache.ignite.internal.index.IndexBuildController.lambda$onPrimaryReplicaElected$9(IndexBuildController.java:320)
> ~[ignite-index-3.1.0-SNAPSHOT.jar:?]
> [13:02:51]W: at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> ~[?:?]
> [13:02:51]W: ... 50 more
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
