Hi all,
we have a problem with flink 1.5.2 high availability in standalone mode.
We have two jobmanagers running. When I shut down the main job manager, the
failover job manager encounters an error during failover.
Logs:
2018-08-17 14:38:16,478 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://fl...@seg-1.adjust.com:29095] has failed, address is now gated for
[50] ms. Reason: [Disassociated]
2018-08-17 14:38:31,449 WARN akka.remote.transport.netty.NettyTransport
- Remote connection to [null] failed with
java.net.ConnectException: Connection refused:
seg-1.adjust.com/178.162.219.66:29095
2018-08-17 14:38:31,451 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://fl...@seg-1.adjust.com:29095] has failed, address is now gated for
[50] ms. Reason: [Association failed with
[akka.tcp://fl...@seg-1.adjust.com:29095]] Caused by: [Connection refused:
seg-1.adjust.com/178.162.219.66:29095]
2018-08-17 14:38:41,379 ERROR
org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler -
Could not retrieve the redirect address.
java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask
timed out on
[Actor[akka.tcp://fl...@seg-1.adjust.com:29095/user/dispatcher#-1599908403]]
after [10000 ms]. Sender[null] sent message of type
"org.apache.flink.runtime.rpc.messages.RemoteFencedMessage".
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
[... shortened ...]
Caused by: akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://fl...@seg-1.adjust.com:29095/user/dispatcher#-1599908403]]
after [10000 ms]. Sender[null] sent message of type
"org.apache.flink.runtime.rpc.messages.RemoteFencedMessage".
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
... 9 more
2018-08-17 14:38:48,005 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
http://seg-2.adjust.com:8083 was granted leadership with
leaderSessionID=708d1a64-c353-448b-9101-7eb3f910970e
2018-08-17 14:38:48,005 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
ResourceManager akka.tcp://fl...@seg-2.adjust.com:30169/user/resourcemanager
was granted leadership with fencing token 8de829de14876a367a80d37194b944ee
2018-08-17 14:38:48,006 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting
the SlotManager.
2018-08-17 14:38:48,007 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher
akka.tcp://fl...@seg-2.adjust.com:30169/user/dispatcher was granted leadership
with fencing token 684f50f8-327c-47e1-a53c-931c4f4ea3e5
2018-08-17 14:38:48,007 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all
persisted jobs.
2018-08-17 14:38:48,021 INFO
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore -
Recovered SubmittedJobGraph(b951bbf518bcf6cc031be6d2ccc441bb, null).
2018-08-17 14:38:48,028 INFO
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore -
Recovered SubmittedJobGraph(06ed64f48fa0a7cffde53b99cbaa073f, null).
2018-08-17 14:38:48,035 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error
occurred in the cluster entrypoint.
java.lang.RuntimeException:
org.apache.flink.runtime.client.JobExecutionException: Could not set up
JobManager
at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:199)
[... shortened ...]
Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set
up JobManager
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:176)
at
org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:936)
at
org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:291)
at
org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:281)
at
org.apache.flink.util.function.ConsumerWithException.accept(ConsumerWithException.java:38)
... 21 more
Caused by: java.lang.Exception: Cannot set up the user code libraries:
/var/lib/flink/ceph/prod/1.5-batch/ha_state/1.5-batch/blob/job_b951bbf518bcf6cc031be6d2ccc441bb/blob_p-a26f62e3bbdcd8884dd18c42a3f6f202b9d2c6e7-0dc87a56862a1f799d515306ffeddfcb
(No such file or directory)
at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:134)
... 25 more
Caused by: java.io.FileNotFoundException:
/var/lib/flink/ceph/prod/1.5-batch/ha_state/1.5-batch/blob/job_b951bbf518bcf6cc031be6d2ccc441bb/blob_p-a26f62e3bbdcd8884dd18c42a3f6f202b9d2c6e7-0dc87a56862a1f799d515306ffeddfcb
(No such file or directory)
at java.io.FileInputStream.open0(Native Method)
[... shortened ...]
... 25 more
2018-08-17 14:38:48,036 INFO org.apache.flink.runtime.blob.TransientBlobCache
- Shutting down BLOB cache
2018-08-17 14:38:48,038 INFO org.apache.flink.runtime.blob.BlobServer
- Stopped BLOB server at 0.0.0.0:27073
Our HA config is:
high-availability: zookeeper
high-availability.cluster-id: 1.5-batch
high-availability.storageDir:
file:///var/lib/flink/ceph//prod/1.5-batch/ha_state
high-availability.zookeeper.path.root: /1.5-batch
high-availability.zookeeper.quorum: kafka-4:2181,kafka-5:2181,kafka-6:2181
Any ideas what might be the probleme here?
Best,
Helmut
