I have increased it to 90000 and seems to be running fine. If I see the failure still when I add some load I will post back in this thread.
On Wed, Jul 14, 2021 at 7:19 PM Debraj Manna <subharaj.ma...@gmail.com> wrote: > Yes I forgot to mention in my first email. I have tried increasing > taskmanager.network.request-backoff.max to > 30000 in flink-conf.yaml. But I am getting the same error. > > On Wed, Jul 14, 2021 at 7:10 PM Timo Walther <twal...@apache.org> wrote: > >> Hi Debraj, >> >> I could find quite a few older emails that were suggesting to play >> around with the `taskmanager.network.request-backoff.max` option. This >> was also recomended in the link that you shared. Have you tried it? >> >> Here is some background: >> >> >> http://deprecated-apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Network-PartitionNotFoundException-when-run-on-multi-nodes-td21683.html >> >> https://issues.apache.org/jira/browse/FLINK-9413 >> >> Let us know if it helped. >> >> Regards, >> Timo >> >> >> >> On 14.07.21 14:51, Debraj Manna wrote: >> > Hi >> > >> > I am observing my flink jobs is failing with the below error >> > >> > 2021-07-14T12:07:00.918Z INFO runtime.executiongraph.Execution >> > flink-akka.actor.default-dispatcher-29 transitionState:1446 >> > MetricAggregateFunction -> (Sink: LateMetricSink10, Sink: TSDBSink9) >> > (12/30) (3489393394c13fd1ad85136e11d67deb) switched from RUNNING to >> > FAILED on container_e12_1626252668574_0001_01_000005 @ platform4 >> > (dataPort=34313). >> > >> > org.apache.flink.runtime.io.network.partition.PartitionNotFoundException: >> Partition >> > 73b5c8ee0c6db91027ce081379f64669#6@7c77e83642e1b954a91dd7b49cf3043b >> not >> > found. >> > >> > at >> > org.apache.flink.runtime.io >> .network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:280) >> > >> > at >> > org.apache.flink.runtime.io >> .network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:198) >> > >> > at >> > org.apache.flink.runtime.io >> .network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:515) >> > >> > at >> > org.apache.flink.runtime.io >> .network.partition.consumer.SingleInputGate.lambda$triggerPartitionStateCheck$1(SingleInputGate.java:845) >> > >> > at >> > >> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) >> > >> > at >> > >> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) >> > >> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) >> > >> > at >> > >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) >> > >> > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> > >> > at >> > >> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> > >> > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> > >> > at >> > >> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> > >> > >> > Same has been discussed in the old link. But in my case there is hardly >> > any load in the cluster. Can someone suggest what can cause this error? >> > >> > >> > >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html >> > < >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html >> > >> > >> > >> > Thanks, >> > >> > >> >>
