[
https://issues.apache.org/jira/browse/FLINK-40388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105209#comment-18105209
]
Avi Sanwal edited comment on FLINK-40388 at 8/17/26 8:30 AM:
-------------------------------------------------------------
Thanks [~hanyou] , that distinction makes sense. I agree that unconditionally
removing the finalizer for every non-running job would be unsafe because the
failure may be temporary and the job may still recover and produce the deletion
savepoint.
We verified the terminal-job behavior in Operator 1.15.0. When the observed job
state is terminal, such as {{{}FAILED{}}}, {{cleanupInternal}} skips savepoint
cleanup and deletes the cluster deployment directly.
One clarification is that this cleanup decision is based on the terminal job
state itself. {{kubernetes.operator.job.restart.failed=false}} allows a failed
job to remain terminal, but the cleanup branch is not directly controlled by
that option.
The case we reproduced does not reach {{{}FAILED{}}}. The resource remains in
the following state:
{code:json}
{
"deletionTimestamp": "<timestamp>",
"finalizers": [
"flinkdeployments.flink.apache.org/finalizer"
],
"jobState": "RESTARTING",
"jmStatus": "MISSING",
"error": null,
"savepointHistoryCount": 0
}
{code}
Because {{RESTARTING}} is not terminal, cleanup selects
{{{}SuspendMode.SAVEPOINT{}}}. {{savepointJobOrError}} then rejects the state
because the job is not {{{}RUNNING{}}}:
{code:java}
CLEANUP | Cleaning up FlinkDeployment
Observing JobManager deployment. Previous status: MISSING
Uncaught error during event processing; another reconciliation will be attempted
cause=java.lang.RuntimeException: Unexpected job status: state=RESTARTING
{code}
The same cleanup error was repeated throughout our bounded observation. The
finalizer remained, {{status.error}} stayed {{{}null{}}}, and no warning event
explained how to recover.
Therefore, the gap appears to be the non-terminal but unrecoverable case: the
JobManager is missing, the recorded job state remains {{{}RESTARTING{}}}, and
the resource cannot progress to either {{RUNNING}} or a terminal state.
We are not proposing immediate or unconditional finalizer removal. A safe
approach could be:
* retain the current retry behavior for transient failures;
* expose a warning event or status condition explaining that deletion is
blocked because the job is not running;
* provide an explicit, configurable fallback for cleanup after a grace period
or lack of progress; and
* keep the default strict if automatic state loss would be unsafe.
Would such a configurable fallback and an observable blocked-deletion condition
be acceptable for this case?
was (Author: JIRAUSER284533):
Thanks [~hanyou], that distinction makes sense. I agree that unconditionally
removing the finalizer for every non-running job would be unsafe because the
failure may be temporary and the job may still recover and produce the deletion
savepoint.
We verified the terminal-job behavior in Operator 1.15.0. When the observed job
state is terminal, such as \{code}FAILED\{code}, \{code}cleanupInternal\{code}
skips savepoint cleanup and deletes the cluster deployment directly.
One clarification is that this cleanup decision is based on the terminal job
state itself. \{code}kubernetes.operator.job.restart.failed=false\{code} allows
a failed job to remain terminal, but the cleanup branch is not directly
controlled by that option.
The case we reproduced does not reach \{code}FAILED\{code}. The resource
remains in the following state:
{code}
{
"deletionTimestamp": "<timestamp>",
"finalizers": [
"flinkdeployments.flink.apache.org/finalizer"
],
"jobState": "RESTARTING",
"jmStatus": "MISSING",
"error": null,
"savepointHistoryCount": 0
}
{code}
Because \{code}RESTARTING\{code} is not terminal, cleanup selects
\{code}SuspendMode.SAVEPOINT\{code}. \{code}savepointJobOrError\{code} then
rejects the state because the job is not \{code}RUNNING\{code}:
{code}
CLEANUP | Cleaning up FlinkDeployment
Observing JobManager deployment. Previous status: MISSING
Uncaught error during event processing; another reconciliation will be attempted
cause=java.lang.RuntimeException: Unexpected job status: state=RESTARTING
{code}
The same cleanup error was repeated throughout our bounded observation. The
finalizer remained, \{code}status.error\{code} stayed \{code}null\{code}, and
no warning event explained how to recover.
Therefore, the gap appears to be the non-terminal but unrecoverable case: the
JobManager is missing, the recorded job state remains \{code}RESTARTING\{code},
and the resource cannot progress to either \{code}RUNNING\{code} or a terminal
state.
We are not proposing immediate or unconditional finalizer removal. A safe
approach could be:
* retain the current retry behavior for transient failures;
* expose a warning event or status condition explaining that deletion is
blocked because the job is not running;
* provide an explicit, configurable fallback for cleanup after a grace period
or lack of progress; and
* keep the default strict if automatic state loss would be unsafe.
Would such a configurable fallback and an observable blocked-deletion condition
be acceptable for this case?
> FlinkDeployment deletion can remain stuck when savepoint-on-deletion is
> enabled and the job is not running
> ----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40388
> URL: https://issues.apache.org/jira/browse/FLINK-40388
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: kubernetes-operator-1.15.0
> Environment: h4. Environment
> * Flink Kubernetes Operator: 1.15.0
> * Apache Flink: 1.20.3
> * Kubernetes: EKS
> h4. Configuration
> {code:yaml}
> spec:
> flinkConfiguration:
> kubernetes.operator.job.savepoint-on-deletion: "true"
> kubernetes.operator.job.drain-on-savepoint-deletion: "true"
> {code}
> Reporter: Avi Sanwal
> Priority: Major
>
> h4. Problem
> Deleting a {{FlinkDeployment}} can remain in {{Terminating}} when
> savepoint-on-deletion is enabled but the Flink job is in a non-running state,
> such as {{RESTARTING}} during recovery or an upgrade.
> h4. Steps to reproduce
> # Deploy a stateful {{FlinkDeployment}} with high availability enabled.
> # Enable {{savepoint-on-deletion}} and {{drain-on-savepoint-deletion}}.
> # Put the job into a non-running state during recovery or an upgrade, for
> example {{RESTARTING}}.
> # Delete the {{FlinkDeployment}} before the job reaches {{RUNNING}}.
> h4. Actual behavior
> The operator attempts deletion-time savepoint handling even though the job
> cannot produce a savepoint. Cleanup is retried while the finalizer remains on
> the {{FlinkDeployment}}, leaving the resource stuck in {{Terminating}} and
> preventing a clean redeployment.
> h4. Expected behavior
> If deletion-time savepoint handling is selected but the observed job is not
> running, the Operator must not retry the same impossible savepoint operation
> indefinitely.
> It should either:
> * complete an explicit, configurable fallback cleanup while preserving
> recoverable HA metadata where available; or
> * retain the resource intentionally, but set an actionable status condition
> and emit a Kubernetes warning event explaining that a savepoint cannot be
> created and what intervention is required.
> h4. Redacted Evidence
> The following sanitized Operator logs show repeated deletion cleanup attempts
> for a non-running job:
> {code}
> [time] INFO CLEANUP | Cleaning up FlinkDeployment
> [time] INFO Observing JobManager deployment. Previous status: MISSING
> [time] INFO JobManager is being deployed
> [time] WARN Uncaught error during event processing; another reconciliation
> will be attempted
> cause=java.lang.RuntimeException:
> Unexpected job status: state=RESTARTING
> [time] INFO CLEANUP | Cleaning up FlinkDeployment
> [time] WARN Uncaught error during event processing
> cause=java.lang.RuntimeException:
> Unexpected job status: state=RESTARTING
> {code}
> The {{FlinkDeployment}} retained
> {{flinkdeployments.flink.apache.org/finalizer}} while {{status.error}}
> remained empty. No Kubernetes warning event explained that deletion was
> blocked by an unavailable savepoint.
> h5. Control case
> With the same deletion settings, deletion completed when the JobManager was
> reachable and the job was running:
> {code}
> [time] INFO CLEANUP | Cleaning up FlinkDeployment
> [time] INFO Job status changed from RESTARTING to RUNNING
> [time] INFO Suspending job with savepoint
> [time] INFO Job successfully suspended with savepoint <savepoint-path>
> [time] INFO DELETED | The resource is deleted
> {code}
> This distinguishes normal savepoint-on-deletion behavior from the failure
> case: the hang occurs when savepoint cleanup is selected for a job that
> cannot produce a savepoint.
> h4. Suggested resolution
> If deletion requests a savepoint but the job is not running, use a safe
> fallback such as *last-state cleanup*:
> * Delete the Flink cluster resources.
> * Retain HA metadata so state recovery remains possible.
> * Record that the deletion-time savepoint was skipped because the job was not
> running.
> * Remove the finalizer.
> The existing drain/savepoint behavior for a running job should remain
> unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
