[
https://issues.apache.org/jira/browse/IGNITE-24614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov updated IGNITE-24614:
---------------------------------------
Reviewer: Mikhail Efremov
> Prevent throwing safe time advance exception on stop
> ----------------------------------------------------
>
> Key: IGNITE-24614
> URL: https://issues.apache.org/jira/browse/IGNITE-24614
> Project: Ignite
> Issue Type: Bug
> Reporter: Vladislav Pyatkov
> Assignee: Vladislav Pyatkov
> Priority: Major
> Labels: ignite-3
> Fix For: 3.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Motivation
> Node might throw {{TrackerClosedException}} an exception if the node was
> going to advise safe time but did not have time to make it:
> {noformat}
> [2025-02-24T15:45:26,966][ERROR][org.apache.ignite.internal.benchmark.MultiTableBenchmark.test-jmh-worker-4][ReplicaManager]
> Could not advance safe time for 429_part_22 to {}
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.util.TrackerClosedException
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
> ~[?:?]
> at
> org.apache.ignite.internal.util.PendingComparableValuesTracker.lambda$cleanupWaitersOnClose$2(PendingComparableValuesTracker.java:192)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
> at
> org.apache.ignite.internal.util.PendingComparableValuesTracker.cleanupWaitersOnClose(PendingComparableValuesTracker.java:192)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.PendingComparableValuesTracker.close(PendingComparableValuesTracker.java:166)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.metastorage.server.time.ClusterTimeImpl.close(ClusterTimeImpl.java:142)
> ~[ignite-metastorage-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.lambda$closeAllManually$1(IgniteUtils.java:617)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
> ~[?:?]
> at
> java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> ~[?:?]
> at
> java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> ~[?:?]
> at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> ~[?:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.closeAllManually(IgniteUtils.java:615)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.closeAllManually(IgniteUtils.java:649)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.metastorage.impl.MetaStorageManagerImpl.stopAsync(MetaStorageManagerImpl.java:772)
> ~[ignite-metastorage-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.lambda$stopAsync$6(IgniteUtils.java:1206)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
> ~[?:?]
> at
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
> ~[?:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.stopAsync(IgniteUtils.java:1212)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.stopAsync(IgniteUtils.java:1254)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.LifecycleManager.initiateAllComponentsStop(LifecycleManager.java:178)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.LifecycleManager.stopNode(LifecycleManager.java:152)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.IgniteImpl.stopAsync(IgniteImpl.java:1610)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.IgniteServerImpl.doShutdownAsync(IgniteServerImpl.java:352)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.IgniteServerImpl.lambda$chainRestartOrShutdownAction$6(IgniteServerImpl.java:291)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
> ~[?:?]
> at
> org.apache.ignite.internal.app.IgniteServerImpl.chainRestartOrShutdownAction(IgniteServerImpl.java:291)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.IgniteServerImpl.shutdownAsync(IgniteServerImpl.java:328)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.app.IgniteServerImpl.shutdown(IgniteServerImpl.java:368)
> ~[ignite-runner-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.lambda$closeAll$0(IgniteUtils.java:570)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> ~[?:?]
> at
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> ~[?:?]
> at
> java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> ~[?:?]
> at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> ~[?:?]
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> ~[?:?]
> at
> java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> ~[?:?]
> at
> org.apache.ignite.internal.util.IgniteUtils.closeAll(IgniteUtils.java:568)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.benchmark.AbstractMultiNodeBenchmark.nodeTearDown(AbstractMultiNodeBenchmark.java:188)
> ~[integrationTest/:?]
> at
> org.apache.ignite.internal.benchmark.jmh_generated.MultiTableBenchmark_test_jmhTest.test_Throughput(MultiTableBenchmark_test_jmhTest.java:103)
> ~[integrationTest/:?]
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> ~[?:?]
> at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
> at
> org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:527)
> ~[jmh-core-1.37.jar:?]
> at
> org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:504)
> ~[jmh-core-1.37.jar:?]
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> ~[?:?]
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> ~[?:?]
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> ~[?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> ~[?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> ~[?:?]
> at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
> Caused by: org.apache.ignite.internal.util.TrackerClosedException
> at
> org.apache.ignite.internal.util.PendingComparableValuesTracker.close(PendingComparableValuesTracker.java:164)
> ~[ignite-core-3.1.0-SNAPSHOT.jar:?]
> ... 59 more
> {noformat}
> This notification can be annoying in case the cluster has several tables with
> many partitions.
> We are already ignoring the exceptions that are typical for stopping:
> {code}
> if (ex != null && !hasCauseOrSuppressed(ex, NodeStoppingException.class)
> && !hasCauseOrSuppressed(ex, CancellationException.class)) {
> LOG.error("Could not advance safe time for {} to {}", ex,
> replica.groupId());
> }
> {code}
> I believe it is a mistake that we are not ignoring TrackerClosedException
> also.
> h3. Definition of done
> * Add a method to close PendingComparableValuesTracker with a specific
> exception and close it with NodeStoppingException.
> * Fixed pattern of message parameters. It exceeds two parameters but takes
> only one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
