[
https://issues.apache.org/jira/browse/FLINK-18263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139146#comment-17139146
]
Mark Cho commented on FLINK-18263:
----------------------------------
Hi [~yunta], [~zhuzh],
In our environment, we typically redeploy Flink jobs using the last available
checkpoints (unless we know that the checkpoint is not compatible with the
redeploy, in which case we use savepoints).
We always enable externalized checkpoints, so for a healthy, running job, we
usually have n checkpoints where n is `state.checkpoints.num-retained`.
We don't typically have jobs that end in `FINISHED` state so we never noticed
this issue before but since we enabled externalized checkpoints, we were not
expecting the JM to delete all the checkpoints on a terminal state.
In this specific job, it was using a Kafka source and used the following method:
{code:java}
KafkaDeserializationSchema::isEndOfStream(T nextElement){code}
It has some custom logic to detect whether to "finish" the current task or not,
and at some point, all Kafka source task hits `isEndOfStream == true` and
requires redeployment with some changed configurations.
For this specific use case, we have different solutions that can address this
job's requirements.
However, we thought the current configuration for ExternalizedCheckpointCleanup
is a bit awkward. When we think about using externalized checkpoints, we would
like to externalize the clean up process of checkpoints for a job, to an
external process. Having the JM manage the clean up process for some state
(like "FINISHED") but not for other states ("CANCELED", "FAILED") seems strange
given that there currently isn't a way to include "FINISHED" state in the
"retain checkpoint" list.
[~zhuzh]'s suggestion on `NEVER_DELETE` or `ALWAYS_RETAIN` is exactly what we
would like. My initial thoughts were that since there is already a config for
retaining on "FAILED" and "FAILED/CANCELED", extending the config to include
"FAILED/CANCELED/FINISHED" would be the cleanest way to achieve this, but
having the config be "Don't Retain" and "Always Retain" is exactly what we
would like to see.
> Allow external checkpoints to be persisted even when the job is in "Finished"
> state.
> ------------------------------------------------------------------------------------
>
> Key: FLINK-18263
> URL: https://issues.apache.org/jira/browse/FLINK-18263
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Reporter: Mark Cho
> Priority: Major
> Labels: pull-request-available
>
> Currently, `execution.checkpointing.externalized-checkpoint-retention`
> configuration supports two options:
> - `DELETE_ON_CANCELLATION` which keeps the externalized checkpoints in FAILED
> and SUSPENDED state.
> - `RETAIN_ON_CANCELLATION` which keeps the externalized checkpoints in
> FAILED, SUSPENDED, and CANCELED state.
> This gives us control over the retention of externalized checkpoints in all
> terminal state of a job, except for the FINISHED state.
> If the job ends up in "FINISHED" state, externalized checkpoints will be
> automatically cleaned up and there currently is no config that will ensure
> that these externalized checkpoints to be persisted.
> I found an old Jira ticket FLINK-4512 where this was discussed. I think it
> would be helpful to have a config that can control the retention policy for
> FINISHED state as well.
> - This can be useful for cases where we want to rewind a job (that reached
> the FINISHED state) to a previous checkpoint.
> - When we use externalized checkpoints, we want to fully delegate the
> checkpoint clean-up to an external process in all job states (without
> cherrypicking FINISHED state to be cleaned up by Flink).
> We have a quick fix working in our fork where we've changed
> `ExternalizedCheckpointCleanup` enum:
> {code:java}
> RETAIN_ON_FAILURE (renamed from DELETE_ON_CANCELLATION; retains on FAILED)
> RETAIN_ON_CANCELLATION (kept the same; retains on FAILED, CANCELED)
> RETAIN_ON_SUCCESS (added; retains on FAILED, CANCELED, FINISHED)
> {code}
> Since this change requires changes to multiple components (e.g. config
> values, REST API, Web UI, etc), I wanted to get the community's thoughts
> before I invest more time in my quick fix PR (which currently only contains
> minimal change to get this working).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
