Zhang Jianguo created FLINK-18480:
-------------------------------------
Summary: JomManager suspend because of loss ZK connection
Key: FLINK-18480
URL: https://issues.apache.org/jira/browse/FLINK-18480
Project: Flink
Issue Type: Bug
Affects Versions: 1.7.2
Reporter: Zhang Jianguo
*Flink version 1.7.2*
*At this time we got network exception and lost connection to zookeeper*
2020-06-29 21:05:39,540 | INFO | [flink-akka.actor.default-dispatcher-601] |
Closing TaskExecutor connection container_e15_1590623517232_44463_01_000002
{color:#de350b}*because: ResourceManager leader changed to new address
null*{color} | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:822) 2020-06-29 21:05:39,540 | INFO |
[flink-akka.actor.default-dispatcher-601] | Closing TaskExecutor connection
container_e15_1590623517232_44463_01_000002 because: ResourceManager leader
changed to new address null | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:822) 2020-06-29 21:05:39,541 | INFO |
[flink-akka.actor.default-dispatcher-494] | Map -> Flat Map -> (Sink:
KafkaParquetSink, Filter -> Map) (12/44) (e8fe39f7a15cce3898ba60c1f422c735)
switched from RUNNING to FAILED. |
org.apache.flink.runtime.executiongraph.ExecutionGraph (Execution.java:1342)
java.lang.Exception: Job leader for job id d42515dde773214b841cef21d1549808
lost leadership. at
org.apache.flink.runtime.taskexecutor.TaskExecutor$JobLeaderListenerImpl.lambda$jobManagerLostLeadership$1(TaskExecutor.java:1526)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:332)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:158)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
*I filterd the change of zk connection state.*
2020-06-29 21:05:39,609 | INFO | [Curator-ConnectionStateManager-0] |
*{color:#de350b}new state: SUSPENDED{color}* |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,611 | INFO | [Curator-ConnectionStateManager-0] |
{color:#de350b}*new state: SUSPENDED*{color} |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,615 | INFO | [Curator-ConnectionStateManager-0] |
*{color:#de350b}new state: SUSPENDED{color}* |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,616 | INFO | [Curator-ConnectionStateManager-0] |
{color:#de350b}*new state: SUSPENDED*{color} |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,710 | INFO | [Curator-ConnectionStateManager-0] | new
state: LOST | org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,725 | INFO | [Curator-ConnectionStateManager-0] | new
state: LOST | org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,731 | INFO | [Curator-ConnectionStateManager-0] | new
state: LOST | org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,736 | INFO | [Curator-ConnectionStateManager-0] |
{color:#de350b}*new state: LOST*{color} |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,933 | INFO | [Curator-ConnectionStateManager-0] | new
state: RECONNECTED | org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,941 | INFO | [Curator-ConnectionStateManager-0] | new
state: RECONNECTED | org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,941 | INFO | [Curator-ConnectionStateManager-0] | new
state: RECONNECTED | org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
2020-06-29 21:05:39,941 | INFO | [Curator-ConnectionStateManager-0] |
{color:#de350b}*new state: RECONNECTED*{color} |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:530)
*After reconnected to ZK, flink tried to recover all jobs.*
2020-06-29 21:05:39,953 | INFO | [flink-akka.actor.default-dispatcher-494] |
Initializing job (d42515dde773214b841cef21d1549808). |
org.apache.flink.runtime.jobmaster.JobMaster (JobMaster.java:271)
2020-06-29 21:05:39,954 | INFO | [flink-akka.actor.default-dispatcher-647] |
ResourceManager akka.ssl.tcp://flink@anawrk00002-3:32586/user/resourcemanager
was granted leadership with fencing token 909783bdcc87da86ee9c7af38ed44f3a |
org.apache.flink.yarn.YarnResourceManager (ResourceManager.java:955)
2020-06-29 21:05:39,954 | INFO | [flink-akka.actor.default-dispatcher-647] |
Starting the SlotManager. |
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager
(SlotManager.java:198)
2020-06-29 21:05:39,983 | INFO | [flink-akka.actor.default-dispatcher-593] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000010
(akka.ssl.tcp://flink@anawrk00015-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,983 | INFO | [flink-akka.actor.default-dispatcher-593] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000005
(akka.ssl.tcp://flink@anawrk00001-3:32327/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,988 | INFO | [flink-akka.actor.default-dispatcher-593] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000009
(akka.ssl.tcp://flink@anawrk00011-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,988 | INFO | [flink-akka.actor.default-dispatcher-593] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000007
(akka.ssl.tcp://flink@anawrk00010-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,990 | INFO | [flink-akka.actor.default-dispatcher-593] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000004
(akka.ssl.tcp://flink@anawrk00006-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,991 | INFO | [flink-akka.actor.default-dispatcher-593] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000008
(akka.ssl.tcp://flink@anawrk00002-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,993 | INFO | [flink-akka.actor.default-dispatcher-601] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000002
(akka.ssl.tcp://flink@anawrk00001-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,993 | INFO | [flink-akka.actor.default-dispatcher-601] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000003
(akka.ssl.tcp://flink@anadat00002-3:32327/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,994 | INFO | [flink-akka.actor.default-dispatcher-601] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000011
(akka.ssl.tcp://flink@anawrk00014-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:39,994 | INFO | [flink-akka.actor.default-dispatcher-494] |
Using restart strategy NoRestartStrategy for
(d42515dde773214b841cef21d1549808). |
org.apache.flink.runtime.jobmaster.JobMaster (JobMaster.java:282)
2020-06-29 21:05:39,995 | INFO | [flink-akka.actor.default-dispatcher-494] |
Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool
at akka://flink/user/59b319a7-00aa-4bf1-b50d-bc7267fe2189 . |
org.apache.flink.runtime.rpc.akka.AkkaRpcService (AkkaRpcService.java:224)
2020-06-29 21:05:40,000 | INFO | [flink-akka.actor.default-dispatcher-494] |
Job recovers via failover strategy: full graph restart |
org.apache.flink.runtime.executiongraph.ExecutionGraph
(ExecutionGraph.java:428)
2020-06-29 21:05:40,001 | INFO | [flink-akka.actor.default-dispatcher-494] |
Running initialization on master for job (d42515dde773214b841cef21d1549808). |
org.apache.flink.runtime.jobmaster.JobMaster (ExecutionGraphBuilder.java:195)
2020-06-29 21:05:40,001 | INFO | [flink-akka.actor.default-dispatcher-494] |
Successfully ran initialization on master in 0 ms. |
org.apache.flink.runtime.jobmaster.JobMaster (ExecutionGraphBuilder.java:224)
2020-06-29 21:05:40,008 | INFO | [flink-akka.actor.default-dispatcher-494] |
Initialized in '/checkpoints/d42515dde773214b841cef21d1549808'. |
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
(ZooKeeperCompletedCheckpointStore.java:137)
2020-06-29 21:05:40,009 | WARN | [flink-akka.actor.default-dispatcher-494] |
Config uses deprecated configuration key 'jobmanager.web.checkpoints.history'
instead of proper key 'web.checkpoints.history' |
org.apache.flink.configuration.Configuration (Configuration.java:797)
2020-06-29 21:05:40,009 | WARN | [flink-akka.actor.default-dispatcher-494] |
Config uses deprecated configuration key 'state.backend.fs.checkpointdir'
instead of proper key 'state.checkpoints.dir' |
org.apache.flink.configuration.Configuration (Configuration.java:797)
2020-06-29 21:05:40,009 | WARN | [flink-akka.actor.default-dispatcher-494] |
Config uses deprecated configuration key 'state.backend.fs.checkpointdir'
instead of proper key 'state.checkpoints.dir' |
org.apache.flink.configuration.Configuration (Configuration.java:797)
2020-06-29 21:05:40,009 | INFO | [flink-akka.actor.default-dispatcher-494] | No
state backend has been configured, using default (Memory / JobManager)
MemoryStateBackend (data in heap memory / checkpoints to JobManager)
(checkpoints: 'hdfs:/tsp/flink/checkpoints', savepoints:
'hdfs:/tsp/flink/savepoint', asynchronous: TRUE, maxStateSize: 5242880) |
org.apache.flink.runtime.jobmaster.JobMaster (StateBackendLoader.java:230)
2020-06-29 21:05:40,035 | INFO | [flink-akka.actor.default-dispatcher-647] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000006
(akka.ssl.tcp://flink@anadat00002-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
2020-06-29 21:05:40,089 | INFO | [flink-akka.actor.default-dispatcher-647] |
Registering TaskManager with ResourceID
container_e15_1590623517232_44463_01_000012
(akka.ssl.tcp://flink@anawrk00005-3:32326/user/taskmanager_0) at
ResourceManager | org.apache.flink.yarn.YarnResourceManager
(ResourceManager.java:730)
...
2020-06-29 21:05:41,978 | INFO | [flink-akka.actor.default-dispatcher-601] |
Cannot serve slot request, no ResourceManager connected. Adding as pending
request [SlotRequestId\{5f26e3fd030369472eb3b9225d97def8}] |
org.apache.flink.runtime.jobmaster.slotpool.SlotPool (SlotPool.java:753)
...
2020-06-29 21:05:41,997 | INFO | [flink-akka.actor.default-dispatcher-494] |
Connecting to ResourceManager
akka.ssl.tcp://flink@anawrk00002-3:32586/user/resourcemanager(909783bdcc87da86ee9c7af38ed44f3a)
| org.apache.flink.runtime.jobmaster.JobMaster (JobMaster.java:1307)
2020-06-29 21:05:41,998 | INFO | [flink-akka.actor.default-dispatcher-647] |
Resolved ResourceManager address, beginning registration |
org.apache.flink.runtime.jobmaster.JobMaster (RetryingRegistration.java:201)
2020-06-29 21:05:41,998 | INFO | [flink-akka.actor.default-dispatcher-647] |
Registration at ResourceManager attempt 1 (timeout=100ms) |
org.apache.flink.runtime.jobmaster.JobMaster (RetryingRegistration.java:250)
2020-06-29 21:05:41,998 | INFO | [flink-akka.actor.default-dispatcher-494] |
Starting ZooKeeperLeaderRetrievalService
/leader/d42515dde773214b841cef21d1549808/job_manager_lock. |
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
(ZooKeeperLeaderRetrievalService.java:100)
2020-06-29 21:05:41,999 | INFO | [flink-akka.actor.default-dispatcher-494] |
Registering job manager
bff6deab98cf97541807aefbd57a4...@akka.ssl.tcp://flink@anawrk00002-3:32586/user/jobmanager_1
for job d42515dde773214b841cef21d1549808. |
org.apache.flink.yarn.YarnResourceManager (ResourceManager.java:295)
2020-06-29 21:05:42,004 | INFO | [flink-akka.actor.default-dispatcher-494] |
Registered job manager
bff6deab98cf97541807aefbd57a4...@akka.ssl.tcp://flink@anawrk00002-3:32586/user/jobmanager_1
for job d42515dde773214b841cef21d1549808. |
org.apache.flink.yarn.YarnResourceManager (ResourceManager.java:676)
2020-06-29 21:05:42,005 | INFO | [flink-akka.actor.default-dispatcher-494] |
JobManager successfully registered at ResourceManager, leader id:
909783bdcc87da86ee9c7af38ed44f3a. |
org.apache.flink.runtime.jobmaster.JobMaster (JobMaster.java:1329)
2020-06-29 21:05:42,005 | INFO | [flink-akka.actor.default-dispatcher-647] |
Requesting new slot [SlotRequestId\{4ea750e557da048edf908efa6ad86564}] and
profile ResourceProfile\{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0,
nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager. |
org.apache.flink.runtime.jobmaster.slotpool.SlotPool (SlotPool.java:709)
Then, maybe network exception happened again.
2020-06-29 21:05:42,615 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 3 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:42,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 3 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:42,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 3 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:42,631 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 3 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:43,616 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 4 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:43,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 4 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:43,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 4 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:43,632 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 4 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:44,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 5 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:44,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 5 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:44,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 5 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:44,632 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 5 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:45,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 6 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:45,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 6 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:45,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 6 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:45,632 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 6 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:46,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 7 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:46,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 7 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:46,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 7 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:46,632 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 7 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:47,617 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 8 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:47,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 8 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:47,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 8 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:47,632 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 8 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:48,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 9 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:48,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 9 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:48,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 9 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:48,632 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 9 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:49,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 10 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:49,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 10 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:49,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 10 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:49,633 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 10 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:50,618 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 11 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:50,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 11 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:50,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 11 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:50,633 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 11 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:51,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 12 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:51,620 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 12 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:51,620 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 12 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:51,633 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 12 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:52,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 13 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:52,620 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 13 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:52,620 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 13 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:52,633 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 13 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:53,619 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 14 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:53,620 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 14 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:53,620 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 14 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:53,634 | INFO | [Suspend state waiting handler] | Connection
to Zookeeper is SUSPENDED. Wait it to be back. Already waited 14 seconds. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:570)
2020-06-29 21:05:54,619 | ERROR | [Suspend state waiting handler] | We've lost
connection to Zookeeper. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:584)
2020-06-29 21:05:54,620 | INFO | [Suspend state waiting handler] |
https://172.28.16.13:32261 lost leadership |
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint
(WebMonitorEndpoint.java:763)
2020-06-29 21:05:54,620 | ERROR | [Suspend state waiting handler] |
*{color:#de350b}We've lost connection to Zookeeper.{color}* |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:584)
2020-06-29 21:05:54,621 | ERROR | [Suspend state waiting handler] | We've lost
connection to Zookeeper. |
org.apache.flink.runtime.leaderelection.SmarterLeaderLatch
(SmarterLeaderLatch.java:584)
{color:#172b4d}*Finally and critically, process suspended here, no more
updating and exception until we found and kill it.*{color}
2020-06-29 21:06:10,709 | INFO | [flink-akka.actor.default-dispatcher-574] |
ResourceManager akka.ssl.tcp://flink@anawrk00002-3:32586/user/resourcemanager
was revoked leadership. Clearing fencing token. |
org.apache.flink.yarn.YarnResourceManager (ResourceManager.java:979)
2020-06-29 21:06:10,709 | INFO | [flink-akka.actor.default-dispatcher-574] |
Stopping ZooKeeperLeaderRetrievalService
/leader/d42515dde773214b841cef21d1549808/job_manager_lock. |
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
(ZooKeeperLeaderRetrievalService.java:117)
2020-06-29 21:06:10,710 | INFO | [flink-akka.actor.default-dispatcher-574] |
*{color:#de350b}Suspending the SlotManager. |
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager
(SlotManager.java:225){color}*
{color:#de350b}*Is there expert fimilar with this situation?*{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
