Any suggestions would be appreciated.
On 2021/10/20 16:18:39, marco <mejrihousse...@gmail.com> wrote:
>
>
> Hello flink community::
>
> I am deploying flink application cluster standalone mode on kubernetes, but i
> am facing some problems
>
> the job starts normally and it continues to run but at some point in time it
> crushes and gets restarted.
>
> Does anyone facing the same problem or know how to resolve it.
>
> I am using flink 1.12.5 version.
>
> here's the log of the JM:
>
> {"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket
> successfully
> opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering
> checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job
> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint
> Timer","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed
> checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in 229
> ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting
> reconnect task to the
> executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling
> reconnect
> task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> for Watch 1899640261","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current
> reconnect backoff is 1000 milliseconds
> (T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
> for Watch 1899640261","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting
> websocket ...
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor
> for Watch 1899640261","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket
> successfully
> opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew
> deadline reached after 120 seconds while renewing lock ConfigMapLock:
> flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping
>
> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> dispatcher
> akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
> all currently running jobs of dispatcher
> akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
> the JobMaster for job
> flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping
>
> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@28d6d80e","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job
> flinkJob (00000000000000000000000000000000) switched from state RUNNING to
> SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
> Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
> org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
>
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
>
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
>
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> akka.actor.Actor.aroundReceive(Actor.scala:517)
> akka.actor.Actor.aroundReceive$(Actor.scala:515)
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> akka.actor.ActorCell.invoke(ActorCell.scala:561)
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> akka.dispatch.Mailbox.run(Mailbox.scala:225)
> akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "}
> {"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source:
> Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source:
> Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
> conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
> conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter
> with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
> with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
> Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
> Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
> Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
> Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd
> call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
> call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
> Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from RUNNING
> to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update
> Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from CANCELING
> to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
> Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> 4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> RUNNING to
> CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
> Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
> CANCELING to
> CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding
> the results produced by task execution
> cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping
> checkpoint coordinator for job
> 00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting
>
> down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job
> 00000000000000000000000000000000 has been
> suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending
>
> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close
> ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping
> JobMaster for job
> flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping
>
> SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect
> job manager
> bc901e3d25d406f70027b81673c74...@akka.tcp://flink@172.24.10.160:6123/user/rpc/jobmanager_2
> for job 00000000000000000000000000000000 from the resource
> manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader
> election
> started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting
> to acquire leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
>
> KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5efd28bd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job
> 00000000000000000000000000000000 reached terminal state
> SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released
> job graph 00000000000000000000000000000000 from
> KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application
> failed unexpectedly:
> ","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
> null
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
>
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal
> error occurred in the cluster
> entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
> Application failed unexpectedly.
>
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
>
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
>
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
>
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
>
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
>
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
>
> org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
>
> java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
>
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.CancellationException: null
> java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
> \t... 22 common frames omitted
> "}
> {"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting
> StandaloneApplicationClusterEntryPoint down with application status UNKNOWN.
> Diagnostics Cluster entrypoint has been closed
> externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
> shutdown hook","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting
> down rest
> endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
> shutdown hook","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped
> BLOB server at
> 0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
> shutdown hook","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully
> Acquired leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting
> to renew leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
> (8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing
> cache directory
> /tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
>
> KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@42c1c89c","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut
> down
> complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
>
> components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
>
> KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4c063eb4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
>
> KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4839a3ec","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
>
> KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@d3b348f","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping
>
> SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing
> the
> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending
> the
> SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
>
> KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop
> thread
> interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
> null
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
>
> io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "}
> {"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@57ae1976","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
>
> DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
>
> KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force
> closing the watch
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The
> watcher is
> closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing
> websocket
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@2a1892aa","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket
> close received. code: 1000, reason:
> ","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
> {"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
> onClose for already closed/closing
> websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
> https://172.31.64.1/...","level":"DEBUG","level_value":10000}
>
>
