The SUSPENDED state is usually caused by lost leadership. Maybe you could find more information about leader in the JobManager and TaskManager logs.
Best, Yang Xiaolong Wang <xiaolong.w...@smartnews.com> 于2022年5月11日周三 19:18写道： > Hello, > > Recently our Flink jobs on Native K8s encountered failing in the > `SUSPENDED` status and got restarted for no reason. > > Flink version: 1.13.2 > > Logs: > ``` > 2022-05-11 05:01:41 > > 2022-05-10 21:01:41,771 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering > checkpoint 17921 (type=CHECKPOINT) @ 1652216501302 for job > 00000000000000000000000000000000.\n > 2022-05-11 05:01:43 > > 2022-05-10 21:01:42,860 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed > checkpoint 17921 for job 00000000000000000000000000000000 (11840 bytes in > 866 ms).\n > 2022-05-11 05:04:34 > > 2022-05-10 21:04:34,550 INFO > org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating a > new watch on TaskManager pods.\n > 2022-05-11 05:06:43 > > 2022-05-10 21:06:43,512 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering > checkpoint 17922 (type=CHECKPOINT) @ 1652216802860 for job > 00000000000000000000000000000000.\n > 2022-05-11 05:06:44 > > 2022-05-10 21:06:44,441 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed > checkpoint 17922 for job 00000000000000000000000000000000 (11840 bytes in > 977 ms).\n > 2022-05-11 05:11:45 > > 2022-05-10 21:11:44,826 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering > checkpoint 17923 (type=CHECKPOINT) @ 1652217104441 for job > 00000000000000000000000000000000.\n > 2022-05-11 05:11:45 > > 2022-05-10 21:11:45,537 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed > checkpoint 17923 for job 00000000000000000000000000000000 (11840 bytes in > 646 ms).\n > 2022-05-11 05:12:36 > > 2022-05-10 21:12:36,746 INFO > org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess > [] - Stopping SessionDispatcherLeaderProcess.\n > 2022-05-11 05:12:36 > > 2022-05-10 21:12:36,747 INFO > org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping > dispatcher akka.tcp://flink@10.2.70.34:6123/user/rpc/dispatcher_1.\n > 2022-05-11 05:12:36 > > 2022-05-10 21:12:36,747 INFO > org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopping all > currently running jobs of dispatcher akka.tcp:// > flink@10.2.70.34:6123/user/rpc/dispatcher_1.\n > 2022-05-11 05:12:36 > > 2022-05-10 21:12:36,749 INFO org.apache.flink.runtime.jobmaster.JobMaster > [] - Stopping the JobMaster for job > insert-into_default_catalog.default_database.sn_fstore_location_cluster_raw_scylla_sink(00000000000000000000000000000000).\n > 2022-05-11 05:12:36 > > 2022-05-10 21:12:36,752 INFO > org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job > 00000000000000000000000000000000 reached terminal state SUSPENDED.\n > 2022-05-11 05:12:36 > > 2022-05-10 21:12:36,752 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job > insert-xxx_sink (00000000000000000000000000000000) switched from state > RUNNING to SUSPENDED.\n > 2022-05-11 05:12:36 > > org.apache.flink.util.FlinkException: Scheduler is being stopped.\n > 2022-05-11 05:12:36 > > at > org.apache.flink.runtime.scheduler.SchedulerBase.closeAsync(SchedulerBase.java:607) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at > org.apache.flink.runtime.jobmaster.JobMaster.stopScheduling(JobMaster.java:962) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at > org.apache.flink.runtime.jobmaster.JobMaster.stopJobExecution(JobMaster.java:926) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:398) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at > org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186) > ~[flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n > 2022-05-11 05:12:36 > > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [flink-dist_2.11-1.13.2.jar:1.13.2]\n" > ... > ``` > > What does the state `SUSPENDED` mean ? And what may possibly cause this > issue ? > > Moreover, I described the jobmanager pod, and got this: > ``` > > Last State: Terminated > > Reason: Error > > Exit Code: 239 > > Started: Tue, 10 May 2022 22:59:42 +0800 > > Finished: Wed, 11 May 2022 05:12:42 +0800 > > ``` > > > Here, what does `Exit Code: 239` mean ? > > Thanks in advanced, > > Yours. >
