Hey Averell, to clarify: You should be able to migrate using a savepoint from 1.10 to 1.11. Restoring from the state stored in Zookeeper (for HA) with a newer Flink version won't work.
On Mon, Oct 26, 2020 at 5:05 PM Robert Metzger <rmetz...@apache.org> wrote: > Hey Averell, > > you should be able to migrate savepoints from Flink 1.10 to 1.11. > > Is there a simple way for me to reproduce this issue locally? This seems > to be a rare, but probably valid issue. Are you using any special > operators? (like the new source API?) > > Best, > Robert > > On Wed, Oct 21, 2020 at 11:07 AM Averell <lvhu...@gmail.com> wrote: > >> Hello Roman, >> >> Thanks for the answer. >> I have already had that high-availability.storageDir configured to an S3 >> location. Our service is not critical enough, so to save the cost, we are >> using the single-master EMR setup. I understand that we'll not get YARN HA >> in that case, but what I expect here is the ability to quickly restore the >> service in the case of failure. Without Zookeeper, when such failure >> happens, I'll need to find the last checkpoint of each job and restore >> from >> there. With the help of HA feature, I can just start a new >> flink-yarn-session, then all jobs will be restored. >> >> I tried to change zookeeper dataDir config to an EFS location which both >> the >> old and new EMR clusters could access, and that worked. >> >> However, now I have a new question: is it expectable to restore to a new >> version of Flink (e.g. saved with Flink1.10 and restored to Flink1.11)? I >> tried and got some error messages attached below. Not sure that's a bug or >> expected behaviour. >> >> Thanks and best regards, >> Averell >> >> ============ >> /07:39:33.906 [main-EventThread] ERROR >> org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState - >> Authentication failed >> 07:40:11.585 [flink-akka.actor.default-dispatcher-2] ERROR >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error >> occurred >> in the cluster entrypoint. >> org.apache.flink.runtime.dispatcher.DispatcherException: Could not start >> recovered job 6e5c12f1c352dd4e6200c40aebb65745. >> at >> >> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:222) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) >> ~[?:1.8.0_265] >> at >> >> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) >> ~[?:1.8.0_265] >> at >> >> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) >> ~[?:1.8.0_265] >> at >> >> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) >> ~[?:1.8.0_265] >> at >> >> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:753) >> ~[?:1.8.0_265] >> at >> >> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) >> ~[?:1.8.0_265] >> at >> >> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.actor.Actor$class.aroundReceive(Actor.scala:517) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.actor.ActorCell.invoke(ActorCell.scala:561) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.dispatch.Mailbox.run(Mailbox.scala:225) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at akka.dispatch.Mailbox.exec(Mailbox.scala:235) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> Caused by: java.util.concurrent.CompletionException: >> org.apache.flink.runtime.client.JobExecutionException: Could not >> instantiate >> JobManager. >> at >> >> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:398) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) >> ~[?:1.8.0_265] >> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> ... 4 more >> Caused by: org.apache.flink.runtime.client.JobExecutionException: Could >> not >> instantiate JobManager. >> at >> >> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:398) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) >> ~[?:1.8.0_265] >> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> ... 4 more >> Caused by: java.lang.NullPointerException >> at >> java.util.Collections$UnmodifiableCollection.<init>(Collections.java:1028) >> ~[?:1.8.0_265] >> at >> java.util.Collections$UnmodifiableList.<init>(Collections.java:1304) >> ~[?:1.8.0_265] >> at java.util.Collections.unmodifiableList(Collections.java:1289) >> ~[?:1.8.0_265] >> at >> >> org.apache.flink.runtime.jobgraph.JobVertex.getOperatorCoordinators(JobVertex.java:352) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:232) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:814) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:228) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.scheduler.SchedulerBase.createExecutionGraph(SchedulerBase.java:269) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:242) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:229) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:119) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:103) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:284) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:272) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.<init>(JobManagerRunnerImpl.java:140) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:84) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$6(Dispatcher.java:388) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) >> ~[?:1.8.0_265] >> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> / >> >> >> >> -- >> Sent from: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >> >
