Hello,
Does anyone have the same issue or have an idea why the jobmanager fails to
renew its leadership when using kubernetes ha service.
Configuration :
kubernetes.namespace: flink-ps-flink-dev
high-availability.kubernetes.leader-election.lease-duration: 200 s
high-availability.kubernetes.leader-election.renew-deadline: 100 s
high-availability.kubernetes.leader-election.retry-period: 15 s
Attached is the log of the error.
Best regards,
{"@timestamp":"2022-03-24T05:53:06.928+02:00","@version":"1","message":"Renew
deadline reached after 100 seconds while renewing lock ConfigMapLock:
flink-ps-flink-dev - elifibre-dispatcher-leader
(1bda7727-ee78-41eb-862e-d0604ae9ccf1)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-11-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2022-03-24T05:53:06.930+02:00","@version":"1","message":"DefaultDispatcherRunner
was revoked the leadership with leader id
444ae619-2d37-4e5b-98aa-a300311959dd. Stopping the
DispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2022-03-24T05:53:06.933+02:00","@version":"1","message":"Stopping
SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
"@timestamp":"2022-03-24T05:53:06.933+02:00","@version":"1","message":"Stopping
dispatcher
akka.tcp://flink@172.24.10.173:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-19","level":"INFO","level_value":20000}
"@timestamp":"2022-03-24T05:53:06.934+02:00","@version":"1","message":"Stopping
all currently running jobs of dispatcher
akka.tcp://flink@172.24.10.173:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-19","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:06.937+02:00","@version":"1","message":"Stopping
the JobMaster for job
lyJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:06.941+02:00","@version":"1","message":"Job
00000000000000000000000000000000 reached terminal state
SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-19","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:06.941+02:00","@version":"1","message":"Job
lyJob (00000000000000000000000000000000) switched from state RUNNING to
SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
Scheduler is being stopped.\n
org.apache.flink.runtime.scheduler.SchedulerBase.closeAsync(SchedulerBase.java:607)\n
org.apache.flink.runtime.jobmaster.JobMaster.stopScheduling(JobMaster.java:962)\n
org.apache.flink.runtime.jobmaster.JobMaster.stopJobExecution(JobMaster.java:926)\n
org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:398)\n
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)\n
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)\n
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)\n
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)\n
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)\n
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)\n
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)\n
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)\n
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\n
akka.actor.Actor$class.arounddeveive(Actor.scala:517)\n
akka.actor.AbstractActor.arounddeveive(AbstractActor.scala:225)\n
akka.actor.ActorCell.deveiveMessage(ActorCell.scala:592)\n
akka.actor.ActorCell.invoke(ActorCell.scala:561)\n
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)\n
akka.dispatch.Mailbox.run(Mailbox.scala:225)\n
akka.dispatch.Mailbox.exec(Mailbox.scala:235)\n
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\n
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\n
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\n"
"@timestamp":"2022-03-24T05:53:07.016+02:00","@version":"1","message":"Source:
Rabbit MQ (1/1) (e03623ee0d7feeac6fa6804e1ca4fb8d) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.018+02:00","@version":"1","message":"Source:
Rabbit MQ (1/1) (e03623ee0d7feeac6fa6804e1ca4fb8d) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.117+02:00","@version":"1","message":"Discarding
the results produced by task execution
e03623ee0d7feeac6fa6804e1ca4fb8d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.120+02:00","@version":"1","message":"Discarding
the results produced by task execution
e03623ee0d7feeac6fa6804e1ca4fb8d.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.121+02:00","@version":"1","message":"Leader
election
started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.122+02:00","@version":"1","message":"Attempting
to acquire leader lease 'ConfigMapLock: flink-ps-flink-dev -
elifibre-dispatcher-leader
(1bda7727-ee78-41eb-862e-d0604ae9ccf1)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.123+02:00","@version":"1","message":"Json
conversion (1/2) (958caae6efc62b04bed501a230ba9faf) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.123+02:00","@version":"1","message":"Json
conversion (1/2) (958caae6efc62b04bed501a230ba9faf) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.123+02:00","@version":"1","message":"Discarding
the results produced by task execution
958caae6efc62b04bed501a230ba9faf.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.123+02:00","@version":"1","message":"Discarding
the results produced by task execution
958caae6efc62b04bed501a230ba9faf.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.124+02:00","@version":"1","message":"Json
conversion (2/2) (e418c3a21e63acb25bc55d71301f3618) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.124+02:00","@version":"1","message":"Json
conversion (2/2) (e418c3a21e63acb25bc55d71301f3618) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.124+02:00","@version":"1","message":"Discarding
the results produced by task execution
e418c3a21e63acb25bc55d71301f3618.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.124+02:00","@version":"1","message":"Discarding
the results produced by task execution
e418c3a21e63acb25bc55d71301f3618.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.124+02:00","@version":"1","message":"Filter
with avro schema (1/2) (9500752fc9511dee8a56978146402346) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.125+02:00","@version":"1","message":"Filter
with avro schema (1/2) (9500752fc9511dee8a56978146402346) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.125+02:00","@version":"1","message":"Discarding
the results produced by task execution
9500752fc9511dee8a56978146402346.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.125+02:00","@version":"1","message":"Discarding
the results produced by task execution
9500752fc9511dee8a56978146402346.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.125+02:00","@version":"1","message":"Filter
with avro schema (2/2) (4ead8748d6ff407b3530558169d01867) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.215+02:00","@version":"1","message":"Filter
with avro schema (2/2) (4ead8748d6ff407b3530558169d01867) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.215+02:00","@version":"1","message":"Discarding
the results produced by task execution
4ead8748d6ff407b3530558169d01867.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.216+02:00","@version":"1","message":"Discarding
the results produced by task execution
4ead8748d6ff407b3530558169d01867.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.216+02:00","@version":"1","message":"Filter
Strategy Pattern (1/2) (175f5ba3b721ee75bd924a82d9920bb3) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.216+02:00","@version":"1","message":"Filter
Strategy Pattern (1/2) (175f5ba3b721ee75bd924a82d9920bb3) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.216+02:00","@version":"1","message":"Discarding
the results produced by task execution
175f5ba3b721ee75bd924a82d9920bb3.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.217+02:00","@version":"1","message":"Discarding
the results produced by task execution
175f5ba3b721ee75bd924a82d9920bb3.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.217+02:00","@version":"1","message":"Filter
Strategy Pattern (2/2) (86f08887f523e34c24ea65a9e530d315) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.217+02:00","@version":"1","message":"Filter
Strategy Pattern (2/2) (86f08887f523e34c24ea65a9e530d315) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.217+02:00","@version":"1","message":"Discarding
the results produced by task execution
86f08887f523e34c24ea65a9e530d315.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.218+02:00","@version":"1","message":"Discarding
the results produced by task execution
86f08887f523e34c24ea65a9e530d315.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.218+02:00","@version":"1","message":"Get
Subscriptions (1/2) (c6c308a2521f91103e12b82216234510) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.218+02:00","@version":"1","message":"Get
Subscriptions (1/2) (c6c308a2521f91103e12b82216234510) switched from CANCELING
to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.218+02:00","@version":"1","message":"Discarding
the results produced by task execution
c6c308a2521f91103e12b82216234510.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.219+02:00","@version":"1","message":"Discarding
the results produced by task execution
c6c308a2521f91103e12b82216234510.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.219+02:00","@version":"1","message":"Get
Subscriptions (2/2) (fa9f1419193985eedcb34650323703e0) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.219+02:00","@version":"1","message":"Get
Subscriptions (2/2) (fa9f1419193985eedcb34650323703e0) switched from CANCELING
to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.219+02:00","@version":"1","message":"Discarding
the results produced by task execution
fa9f1419193985eedcb34650323703e0.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.219+02:00","@version":"1","message":"Discarding
the results produced by task execution
fa9f1419193985eedcb34650323703e0.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.220+02:00","@version":"1","message":"FDR call
(1/2) (857df6df13fd25fa685338498e5a56df) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.220+02:00","@version":"1","message":"FDR call
(1/2) (857df6df13fd25fa685338498e5a56df) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.220+02:00","@version":"1","message":"Discarding
the results produced by task execution
857df6df13fd25fa685338498e5a56df.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.220+02:00","@version":"1","message":"Discarding
the results produced by task execution
857df6df13fd25fa685338498e5a56df.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.220+02:00","@version":"1","message":"FDR call
(2/2) (c2fb784a3724bd2e41681d894edfb62a) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.221+02:00","@version":"1","message":"FDR call
(2/2) (c2fb784a3724bd2e41681d894edfb62a) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.221+02:00","@version":"1","message":"Discarding
the results produced by task execution
c2fb784a3724bd2e41681d894edfb62a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.221+02:00","@version":"1","message":"Discarding
the results produced by task execution
c2fb784a3724bd2e41681d894edfb62a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.221+02:00","@version":"1","message":"Update
Main devord (1/2) (06bda7e24f422db2706937dcf8fbec84) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.221+02:00","@version":"1","message":"Update
Main devord (1/2) (06bda7e24f422db2706937dcf8fbec84) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.222+02:00","@version":"1","message":"Discarding
the results produced by task execution
06bda7e24f422db2706937dcf8fbec84.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.222+02:00","@version":"1","message":"Discarding
the results produced by task execution
06bda7e24f422db2706937dcf8fbec84.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.222+02:00","@version":"1","message":"Update
Main devord (2/2) (d7cc237ba04dbe5dbcd8019fb9e9f37a) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.222+02:00","@version":"1","message":"Update
Main devord (2/2) (d7cc237ba04dbe5dbcd8019fb9e9f37a) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.226+02:00","@version":"1","message":"Successfully
Acquired leader lease 'ConfigMapLock: flink-ps-flink-dev -
elifibre-dispatcher-leader
(1bda7727-ee78-41eb-862e-d0604ae9ccf1)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-9874-thread-1","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.315+02:00","@version":"1","message":"Discarding
the results produced by task execution
d7cc237ba04dbe5dbcd8019fb9e9f37a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.315+02:00","@version":"1","message":"DefaultDispatcherRunner
was granted leadership with leader id 249a7d14-e43d-44c5-b7c6-5ed4a38787ec.
Creating new
DispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.316+02:00","@version":"1","message":"Discarding
the results produced by task execution
d7cc237ba04dbe5dbcd8019fb9e9f37a.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.316+02:00","@version":"1","message":"Sink:
Send to Erable (1/2) (9bcd2a875774cd7b9b8e632e012dedc6) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.316+02:00","@version":"1","message":"Sink:
Send to Erable (1/2) (9bcd2a875774cd7b9b8e632e012dedc6) switched from CANCELING
to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.316+02:00","@version":"1","message":"Attempting
to renew leader lease 'ConfigMapLock: flink-ps-flink-dev -
elifibre-dispatcher-leader
(1bda7727-ee78-41eb-862e-d0604ae9ccf1)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.319+02:00","@version":"1","message":"Discarding
the results produced by task execution
9bcd2a875774cd7b9b8e632e012dedc6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.319+02:00","@version":"1","message":"deveived
resource requirements from job 00000000000000000000000000000000:
[ResourceRequirementresourceProfile=ResourceProfileUNKNOWN,
numberOfRequiredSlots=1]","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager","thread_name":"flink-akka.actor.default-dispatcher-19","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.320+02:00","@version":"1","message":"Sink:
Send to HDBSINK (2/2) (feb012a4a3da6bd023b64b941f22e3f8) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.320+02:00","@version":"1","message":"Sink:
Send to HDBSINK (2/2) (feb012a4a3da6bd023b64b941f22e3f8) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.320+02:00","@version":"1","message":"Discarding
the results produced by task execution
feb012a4a3da6bd023b64b941f22e3f8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.321+02:00","@version":"1","message":"Clearing
resource requirements of job
00000000000000000000000000000000","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager","thread_name":"flink-akka.actor.default-dispatcher-19","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.321+02:00","@version":"1","message":"Stopping
checkpoint coordinator for job
00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.321+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.321+02:00","@version":"1","message":"Shutting
down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.321+02:00","@version":"1","message":"Job
00000000000000000000000000000000 has been
suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.420+02:00","@version":"1","message":"Releasing
slot
[0a4a4997a7b8cc75929635725d36b45c].","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.421+02:00","@version":"1","message":"Releasing
slot
[95c7d3c0143981528b0896028135594c].","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.421+02:00","@version":"1","message":"Close
ResourceManager connection d620bbfd7f501bfd64a6ecb431e7dc84: Stopping JobMaster
for job
lyJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.422+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.422+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriverconfigMapName='elifibre-resourcemanager-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.422+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3dc63276","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.422+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.422+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1a1e3ed1","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.422+02:00","@version":"1","message":"Disconnect
job manager
b9181123fcb3f7d8f140b321cddb4...@akka.tcp://flink@172.24.10.173:6123/user/rpc/jobmanager_2
for job 00000000000000000000000000000000 from the resource
manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-19","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.423+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.515+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.519+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.519+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriverconfigMapName='elifibre-00000000000000000000000000000000-jobmanager-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.519+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@26803820","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.519+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.520+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@2e180a94","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.520+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)\n
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)\n
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
java.lang.Thread.run(Thread.java:748)\n"
"@timestamp":"2022-03-24T05:53:07.520+02:00","@version":"1","message":"Released
job graph 00000000000000000000000000000000 from
KubernetesStateHandleStoreconfigMapName='elifibre-dispatcher-leader'.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-3","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.520+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.521+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.616+02:00","@version":"1","message":"Application
failed unexpectedly:
","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-3","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
null\n
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\n
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)\n
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:273)\n
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)\n
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)\n
org.apache.flink.runtime.concurrent.DidevtExecutorService.execute(DidevtExecutorService.java:217)\n
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)\n
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)\n
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)\n
org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:991)\n
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)\n
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)\n
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)\n
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
java.lang.Thread.run(Thread.java:748)\n"
"@timestamp":"2022-03-24T05:53:07.618+02:00","@version":"1","message":"Fatal
error occurred in the cluster
entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-3","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
Application failed unexpectedly.\n
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)\n
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)\n
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)\n
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)\n
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)\n
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:273)\n
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)\n
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)\n
org.apache.flink.runtime.concurrent.DidevtExecutorService.execute(DidevtExecutorService.java:217)\n
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)\n
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)\n
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)\n
org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:991)\n
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)\n
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)\n
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)\n
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)\n
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)\n
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)\n
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
java.lang.Thread.run(Thread.java:748)\nCaused by:
java.util.concurrent.CancellationException: null\n
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)\n\t...
22 common frames omitted\n"
"@timestamp":"2022-03-24T05:53:07.622+02:00","@version":"1","message":"Shutting
StandaloneApplicationClusterEntryPoint down with application status UNKNOWN.
Diagnostics Cluster entrypoint has been closed
externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
shutdown hook","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.623+02:00","@version":"1","message":"Stopped
BLOB server at
0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
shutdown hook","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.624+02:00","@version":"1","message":"Shutting
down rest
endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
shutdown hook","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.818+02:00","@version":"1","message":"Removing
cache didevtory
/tmp/flink-web-1c1f0f2b-8e43-4367-adf8-212af8f9ff2a/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.819+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.819+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriverconfigMapName='elifibre-restserver-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.819+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@1ba98abf","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.820+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.820+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@50364cef","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.820+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)\n
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)\n
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
java.lang.Thread.run(Thread.java:748)\n"
"@timestamp":"2022-03-24T05:53:07.820+02:00","@version":"1","message":"Shut
down
complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.820+02:00","@version":"1","message":"Closing
components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.821+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.821+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.821+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriverconfigMapName='elifibre-dispatcher-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.821+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.821+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@2d065ec2","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.821+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5ef1b899","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriverconfigMapName='elifibre-resourcemanager-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@59ca2253","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5ace279f","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.822+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriverconfigMapName='elifibre-dispatcher-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@46ae92ca","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)\n
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)\n
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
java.lang.Thread.run(Thread.java:748)\n"
"@timestamp":"2022-03-24T05:53:07.823+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@1fccb2a2","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-42","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.915+02:00","@version":"1","message":"Stopping
SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-42","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.916+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.916+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.917+02:00","@version":"1","message":"Closing
the slot
manager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.917+02:00","@version":"1","message":"Suspending
the slot
manager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriverconfigMapName='elifibre-resourcemanager-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@2586005d","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@3de47758","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)\n
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)\n
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)\n
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)\n
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
java.lang.Thread.run(Thread.java:748)\n"
"@timestamp":"2022-03-24T05:53:07.918+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.919+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriverconfigMapName='elifibre-00000000000000000000000000000000-jobmanager-leader'.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.919+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.919+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@64b13ca9","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.920+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.920+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-13","level":"INFO","level_value":20000
"@timestamp":"2022-03-24T05:53:07.920+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@759b483","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-13","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.921+02:00","@version":"1","message":"WebSocket
close deveived. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
"@timestamp":"2022-03-24T05:53:07.921+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000
