For native K8s integration, the Flink ResourceManager will delete the
JobManager K8s deployment as well as the HA data once the job reached a
globally terminal state.

However, it is indeed a problem for standalone mode since the JobManager
will be restarted again even the job has finished. I think the
flink-kubernetes-operator could handle this situation by doing the cleanup.


Best,
Yang

Austin Cawley-Edwards <austin.caw...@gmail.com> 于2022年9月8日周四 06:01写道:

> Hey Gil,
>
> I'm referring to when a pod exits on its own, not when being deleted.
> Deployments only support the "Always" restart policy [1].
>
> In my understanding, the JM only cleans up HA data when it is shutdown[2],
> after which the process will exit which leads to the problem with k8s
> Deployment restart policies.
>
> Best,
> Austin
>
> [1]:
> https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pod-template
> [2]:
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/rest_api/#cluster
>
> On Wed, Sep 7, 2022 at 4:43 PM Gil De Grove <gil.degr...@euranova.eu>
> wrote:
>
>> Hello Austin,
>>
>> I'm not aware of any limitations of deployement not letting pod exit
>> (correctly or incorrectly). What do you mean by that exactly? Would it be
>> possible for you to point out to piece of documentation that make you think
>> that ?
>>
>> A pod, if correctly setup will be exited when receiving it's sigterm or
>> sigkill from the orchestrator.
>> So when "deleting" the deployment, the pods are quitted correctly. In the
>> case flink did triggered a savepoint before, you can then restart from that
>> savepoint.
>> Usually, when a pod is not being terminated this means that the SIG is
>> not transferred to the correct process.
>>
>> Hopes this helps.
>>
>> Regards,
>> Gil
>>
>>
>> On Wed, Sep 7, 2022, 21:16 Austin CawleyEdwards <austin.caw...@gmail.com>
>> wrote:
>>
>>> Cool, thanks! How does it clean up the HA data, if the cluster is never
>>> able to shut down (due to the k8s Deployment restriction)?
>>>
>>> Best,
>>> Austin
>>>
>>> On Mon, Sep 5, 2022 at 6:51 PM Gyula Fóra <gyula.f...@gmail.com> wrote:
>>>
>>>> Hi!
>>>>
>>>> The operator supports both Flink native and standalone deployment modes
>>>> and in both cases the JM is deployed as k8s Deployment.
>>>>
>>>> During upgrade Flink/operator deletes the deployment after savepoint
>>>> and waits for termination before it creates a new one with the updated
>>>> spec.
>>>>
>>>> Cheers,
>>>> Gyula
>>>>
>>>> On Mon, 5 Sep 2022 at 07:41, Austin Cawley-Edwards <
>>>> austin.caw...@gmail.com> wrote:
>>>>
>>>>> Hey Marco,
>>>>>
>>>>> Unfortunately there is no built in k8s API that models an application
>>>>> mode JM exactly but Deployments should be fine, in general. As Gyula 
>>>>> notes,
>>>>> where they can be difficult is during application upgrades as Deployments
>>>>> never let their pods exit, even if successful, so there is no way to stop
>>>>> the cluster gracefully.
>>>>>
>>>>> Is stopping your application with a savepoint and redeploying a
>>>>> workable solution for image upgrades? In this way a Job could still be
>>>>> used.
>>>>>
>>>>>
>>>>> @Gyula, how are JMs handled in the operator? Job, Deployment, or
>>>>> something custom?
>>>>>
>>>>>
>>>>> Best,
>>>>> Austin
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 5, 2022 at 6:15 AM Gyula Fóra <gyula.f...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> You can use deployments of course , the operator and native k8s
>>>>>> integration does exactly that.
>>>>>>
>>>>>> Even then job updates can be tricky so I believe you are much better
>>>>>> off with the operator.
>>>>>>
>>>>>> Gyula
>>>>>>
>>>>>> On Sun, 4 Sep 2022 at 11:11, marco andreas <marcoandreas...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Thanks for the response, I will take a look at it.
>>>>>>>
>>>>>>> But if we aren't able to use the flink operator due to technical
>>>>>>> constraints is it possible to deploy the JM as deployment without any
>>>>>>> consequences that I am not aware of?
>>>>>>>
>>>>>>> Sincerely,
>>>>>>>
>>>>>>> Le sam. 3 sept. 2022 à 23:27, Gyula Fóra <gyula.f...@gmail.com> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi!
>>>>>>>> You should check out the Flink Kubernetes Operator. I think that
>>>>>>>> covers all your needs .
>>>>>>>>
>>>>>>>>
>>>>>>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Gyula
>>>>>>>>
>>>>>>>> On Sat, 3 Sep 2022 at 13:45, marco andreas <
>>>>>>>> marcoandreas...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> We are deploying a flink application cluster on k8S. Following the
>>>>>>>>> official documentation the JM is deployed As a job resource , however 
>>>>>>>>> we
>>>>>>>>> are deploying a long running flink job that is not supposed to be
>>>>>>>>> terminated and also we need to update the image of the flink job.
>>>>>>>>>
>>>>>>>>>  The problem is that the job is an immutable resource, we
>>>>>>>>> cant update it.
>>>>>>>>>
>>>>>>>>> So I'm wondering if it's possible to use a deployment resource for
>>>>>>>>> the jobmanager and if there will be any side effects or repercussions.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>> Gil De Grove
>>
>

Reply via email to