For native K8s integration, the Flink ResourceManager will delete the JobManager K8s deployment as well as the HA data once the job reached a globally terminal state.
However, it is indeed a problem for standalone mode since the JobManager will be restarted again even the job has finished. I think the flink-kubernetes-operator could handle this situation by doing the cleanup. Best, Yang Austin Cawley-Edwards <austin.caw...@gmail.com> 于2022年9月8日周四 06:01写道： > Hey Gil, > > I'm referring to when a pod exits on its own, not when being deleted. > Deployments only support the "Always" restart policy [1]. > > In my understanding, the JM only cleans up HA data when it is shutdown[2], > after which the process will exit which leads to the problem with k8s > Deployment restart policies. > > Best, > Austin > > [1]: > https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pod-template > [2]: > https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/rest_api/#cluster > > On Wed, Sep 7, 2022 at 4:43 PM Gil De Grove <gil.degr...@euranova.eu> > wrote: > >> Hello Austin, >> >> I'm not aware of any limitations of deployement not letting pod exit >> (correctly or incorrectly). What do you mean by that exactly? Would it be >> possible for you to point out to piece of documentation that make you think >> that ? >> >> A pod, if correctly setup will be exited when receiving it's sigterm or >> sigkill from the orchestrator. >> So when "deleting" the deployment, the pods are quitted correctly. In the >> case flink did triggered a savepoint before, you can then restart from that >> savepoint. >> Usually, when a pod is not being terminated this means that the SIG is >> not transferred to the correct process. >> >> Hopes this helps. >> >> Regards, >> Gil >> >> >> On Wed, Sep 7, 2022, 21:16 Austin CawleyEdwards <austin.caw...@gmail.com> >> wrote: >> >>> Cool, thanks! How does it clean up the HA data, if the cluster is never >>> able to shut down (due to the k8s Deployment restriction)? >>> >>> Best, >>> Austin >>> >>> On Mon, Sep 5, 2022 at 6:51 PM Gyula Fóra <gyula.f...@gmail.com> wrote: >>> >>>> Hi! >>>> >>>> The operator supports both Flink native and standalone deployment modes >>>> and in both cases the JM is deployed as k8s Deployment. >>>> >>>> During upgrade Flink/operator deletes the deployment after savepoint >>>> and waits for termination before it creates a new one with the updated >>>> spec. >>>> >>>> Cheers, >>>> Gyula >>>> >>>> On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards < >>>> austin.caw...@gmail.com> wrote: >>>> >>>>> Hey Marco, >>>>> >>>>> Unfortunately there is no built in k8s API that models an application >>>>> mode JM exactly but Deployments should be fine, in general. As Gyula >>>>> notes, >>>>> where they can be difficult is during application upgrades as Deployments >>>>> never let their pods exit, even if successful, so there is no way to stop >>>>> the cluster gracefully. >>>>> >>>>> Is stopping your application with a savepoint and redeploying a >>>>> workable solution for image upgrades? In this way a Job could still be >>>>> used. >>>>> >>>>> >>>>> @Gyula, how are JMs handled in the operator? Job, Deployment, or >>>>> something custom? >>>>> >>>>> >>>>> Best, >>>>> Austin >>>>> >>>>> >>>>> >>>>> On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra <gyula.f...@gmail.com> >>>>> wrote: >>>>> >>>>>> You can use deployments of course , the operator and native k8s >>>>>> integration does exactly that. >>>>>> >>>>>> Even then job updates can be tricky so I believe you are much better >>>>>> off with the operator. >>>>>> >>>>>> Gyula >>>>>> >>>>>> On Sun, 4 Sep 2022 at 11:11, marco andreas <marcoandreas...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> Thanks for the response, I will take a look at it. >>>>>>> >>>>>>> But if we aren't able to use the flink operator due to technical >>>>>>> constraints is it possible to deploy the JM as deployment without any >>>>>>> consequences that I am not aware of? >>>>>>> >>>>>>> Sincerely, >>>>>>> >>>>>>> Le sam. 3 sept. 2022 à 23:27, Gyula Fóra <gyula.f...@gmail.com> a >>>>>>> écrit : >>>>>>> >>>>>>>> Hi! >>>>>>>> You should check out the Flink Kubernetes Operator. I think that >>>>>>>> covers all your needs . >>>>>>>> >>>>>>>> >>>>>>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Gyula >>>>>>>> >>>>>>>> On Sat, 3 Sep 2022 at 13:45, marco andreas < >>>>>>>> marcoandreas...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> We are deploying a flink application cluster on k8S. Following the >>>>>>>>> official documentation the JM is deployed As a job resource , however >>>>>>>>> we >>>>>>>>> are deploying a long running flink job that is not supposed to be >>>>>>>>> terminated and also we need to update the image of the flink job. >>>>>>>>> >>>>>>>>> The problem is that the job is an immutable resource, we >>>>>>>>> cant update it. >>>>>>>>> >>>>>>>>> So I'm wondering if it's possible to use a deployment resource for >>>>>>>>> the jobmanager and if there will be any side effects or repercussions. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>> Gil De Grove >> >
