Hello,
Recently our Flink jobs on Native K8s encountered failing in the
`SUSPENDED` status and got restarted for no reason.
Flink version: 1.13.2
Logs:
```
2022-05-11 05:01:41
2022-05-10 21:01:41,771 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 17921 (type=CHECKPOINT) @ 1652216501302 for job
00000000000000000000000000000000.\n
2022-05-11 05:01:43
2022-05-10 21:01:42,860 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 17921 for job 00000000000000000000000000000000 (11840 bytes in
866 ms).\n
2022-05-11 05:04:34
2022-05-10 21:04:34,550 INFO
org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating a
new watch on TaskManager pods.\n
2022-05-11 05:06:43
2022-05-10 21:06:43,512 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 17922 (type=CHECKPOINT) @ 1652216802860 for job
00000000000000000000000000000000.\n
2022-05-11 05:06:44
2022-05-10 21:06:44,441 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 17922 for job 00000000000000000000000000000000 (11840 bytes in
977 ms).\n
2022-05-11 05:11:45
2022-05-10 21:11:44,826 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 17923 (type=CHECKPOINT) @ 1652217104441 for job
00000000000000000000000000000000.\n
2022-05-11 05:11:45
2022-05-10 21:11:45,537 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 17923 for job 00000000000000000000000000000000 (11840 bytes in
646 ms).\n
2022-05-11 05:12:36
2022-05-10 21:12:36,746 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
[] - Stopping SessionDispatcherLeaderProcess.\n
2022-05-11 05:12:36
2022-05-10 21:12:36,747 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping
dispatcher akka.tcp://flink@10.2.70.34:6123/user/rpc/dispatcher_1.\n
2022-05-11 05:12:36
2022-05-10 21:12:36,747 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping all
currently running jobs of dispatcher akka.tcp://
flink@10.2.70.34:6123/user/rpc/dispatcher_1.\n
2022-05-11 05:12:36
2022-05-10 21:12:36,749 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Stopping the JobMaster for job
insert-into_default_catalog.default_database.sn_fstore_location_cluster_raw_scylla_sink(00000000000000000000000000000000).\n
2022-05-11 05:12:36
2022-05-10 21:12:36,752 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
00000000000000000000000000000000 reached terminal state SUSPENDED.\n
2022-05-11 05:12:36
2022-05-10 21:12:36,752 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
insert-xxx_sink (00000000000000000000000000000000) switched from state
RUNNING to SUSPENDED.\n
2022-05-11 05:12:36
org.apache.flink.util.FlinkException: Scheduler is being stopped.\n
2022-05-11 05:12:36
at
org.apache.flink.runtime.scheduler.SchedulerBase.closeAsync(SchedulerBase.java:607)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at
org.apache.flink.runtime.jobmaster.JobMaster.stopScheduling(JobMaster.java:962)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at
org.apache.flink.runtime.jobmaster.JobMaster.stopJobExecution(JobMaster.java:926)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:398)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
~[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n
2022-05-11 05:12:36
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.13.2.jar:1.13.2]\n"
...
```
What does the state `SUSPENDED` mean ? And what may possibly cause this
issue ?
Moreover, I described the jobmanager pod, and got this:
```
Last State: Terminated
Reason: Error
Exit Code: 239
Started: Tue, 10 May 2022 22:59:42 +0800
Finished: Wed, 11 May 2022 05:12:42 +0800
```
Here, what does `Exit Code: 239` mean ?
Thanks in advanced,
Yours.
