Any update on this from community side? On Tue, May 7, 2019 at 6:43 PM Manjusha Vuyyuru <vmanjusha....@gmail.com> wrote:
> im using 1.7.2. > > > On Tue, May 7, 2019 at 5:50 PM miki haiat <miko5...@gmail.com> wrote: > >> Which flink version are you using? >> I had similar issues with 1.5.x >> >> On Tue, May 7, 2019 at 2:49 PM Manjusha Vuyyuru <vmanjusha....@gmail.com> >> wrote: >> >>> Hello, >>> >>> I have a flink setup with two job managers coordinated by zookeeper. >>> >>> I see the below exception and both jobmanagers are going down: >>> >>> 2019-05-07 08:29:13,346 INFO >>> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore - >>> Released locks of job graph f8eb1b482d8ec8c1d3e94c4d0f79df77 from ZooKeeper. >>> 2019-05-07 08:29:13,346 ERROR >>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint -* Fatal >>> error occurred in the cluster entrypoint.* >>> java.lang.RuntimeException: org.apache.flink.util.FlinkException: Could >>> not retrieve submitted JobGraph from state handle under >>> /147dd022ec91f7381ad4ca3d290387e9. This indicates that the retrieved state >>> handle is broken. Try cleaning the state handle store. >>> at >>> org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:199) >>> at >>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:74) >>> at >>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) >>> at >>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) >>> at >>> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) >>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) >>> at >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) >>> at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> at >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> Caused by: org.apache.flink.util.FlinkException: Could not retrieve >>> submitted JobGraph from state handle under >>> /147dd022ec91f7381ad4ca3d290387e9. This indicates that the retrieved state >>> handle is broken. Try cleaning the state handle store. >>> at >>> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.recoverJobGraph(ZooKeeperSubmittedJobGraphStore.java:208) >>> at >>> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJob(Dispatcher.java:696) >>> at >>> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobGraphs(Dispatcher.java:681) >>> at >>> org.apache.flink.runtime.dispatcher.Dispatcher.recoverJobs(Dispatcher.java:662) >>> at >>> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$26(Dispatcher.java:821) >>> at >>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:72) >>> ... 9 more >>> >>> >>> Can someone please help me understand in detail on what is causing this >>> exception. I can see zookeeper not able to retrieve job graph. What could >>> be the reason for this? >>> >>> This is second time that my setup is going down with this excepton, >>> first time i cleared jobgraph folder in zookeeper and restarted, now again >>> faced with same issue. >>> >>> Since this is production setup this way of outage is not at all expected >>> :(. Can someone help me how to give a permanent fix to this issue? >>> >>> >>> Thanks, >>> Manju >>> >>>
