Noa Ramon created FLINK-41404:
---------------------------------
Summary: Blue/Green: after an aborted transition the retry
restores the aborted attempt's state instead of the serving deployment's fresh
savepoint
Key: FLINK-41404
URL: https://issues.apache.org/jira/browse/FLINK-41404
Project: Flink
Issue Type: Bug
Affects Versions: kubernetes-operator-1.15.0
Reporter: Noa Ramon
When a two-deployment Blue/Green transition aborts,
\{{BlueGreenDeploymentService#abortDeployment}} suspends the new child
(\{{BlueGreenDeploymentService.java:685}} on \{{main}} at 43719c98) and rolls
the parent back to the previous \{{ACTIVE_*}} state. The next transition
re-creates the child under the same name with \{{createOrReplace}}
(\{{BlueGreenKubernetesService.java:59}}). Because the existing resource keeps
its status, the \{{FlinkDeployment}} controller treats this as a *resume of a
suspended deployment*, not as a first deployment.
\{{spec.job.initialSavepointPath}}, which the Blue/Green controller sets to the
serving deployment's fresh savepoint, is only honoured on a first deployment.
The retry therefore restores the aborted attempt's state in both stateful modes:
# *\{{upgradeMode: last-state}}.* The suspend keeps the Kubernetes HA metadata
(\{{SuspendMode#deleteHaMeta}} is \{{false}} only for \{{LAST_STATE}}). On
resume the JobManager recovers the aborted attempt's latest checkpoint from the
HA ConfigMaps before \{{upgradeMode}} or \{{initialSavepointPath}} take effect.
Switching the spec to \{{stateless}} does not help, because the HA restore
happens first.
# *\{{upgradeMode: savepoint}}.* The suspend takes a stop-savepoint of the
aborted child and records it in \{{status.jobStatus.upgradeSavepointPath}}. On
resume the job restores from that savepoint.
\{{AbstractJobReconciler#restoringFromInitialSavepoint}} does not apply,
because the new \{{initialSavepointPath}} differs from the recorded
\{{upgradeSavepointPath}}.
*Impact:* if an attempt aborts because of its state (for example a
\{{StateMigrationException}} after an incompatible serializer change), every
later attempt on that colour fails the same way. The serving deployment's good
savepoint is never used. The only way out is deleting the child
\{{FlinkDeployment}} by hand.
h4. Observed on 1.15.0
* \{{last-state}}: a transition to a new build restored the previous savepoint,
failed with \{{StateMigrationException}} and aborted (expected). A follow-up
transition with \{{upgradeMode: stateless}} and no savepoint path still logged
\{{Restoring SplitEnumerator ... from checkpoint}} with the checkpoint taken
from the leftover HA ConfigMaps, and failed with the same exception.
* \{{savepoint}}: after a forced abort and a retry, the parent's child spec and
the JobManager disagreed:
** \{{spec.job.initialSavepointPath = .../savepoint-e28232-...}} (the serving
deployment's fresh savepoint)
** JobManager \{{execution.savepoint.path = .../savepoint-8b5a03-...}} (the
aborted child's own stop-savepoint)
h4. Steps to reproduce
# Deploy a \{{FlinkBlueGreenDeployment}} with \{{upgradeMode: savepoint}} (or
\{{last-state}}) and wait for \{{ACTIVE_BLUE}}.
# Set \{{kubernetes.operator.bluegreen.abort.grace-period}} lower than the time
the job needs to become ready (a few seconds), then make any spec change. The
transition aborts, \{{<name>-green}} is suspended, and the parent returns to
\{{ACTIVE_BLUE}}.
# Restore a normal grace period and make another valid spec change.
\{{<name>-green}} is re-created under the same name.
# Expected: \{{<name>-green}} starts from blue's fresh savepoint
(\{{spec.job.initialSavepointPath}}). Actual: it resumes from its own suspend
savepoint (\{{savepoint}} mode) or from its last checkpoint via HA
(\{{last-state}} mode).
h4. Proposed fix
In a two-deployment transition, delete the aborted child instead of suspending
it, the same way a successful cutover deletes the outgoing deployment. Deletion
goes through \{{ApplicationReconciler#cleanupInternal}}: a running job is
cancelled with \{{SuspendMode.STATELESS}} (or \{{SAVEPOINT}} when
savepoint-on-deletion is enabled), and both delete the HA metadata. The child's
status is removed together with the resource, so the next transition creates a
fresh resource that honours \{{initialSavepointPath}}. An abort of the *first*
deployment, where there is no other deployment to fall back to, keeps the
current suspend behaviour.
Trade-off: the aborted deployment's pods are no longer kept around for
inspection. The parent still records the reason in \{{status.error}} and the
operator logs the abort.
h4. Verification
* New regression test
\{{verifyRetryAfterAbortRestoresTheServingColoursFreshSavepoint}}: aborts a
\{{savepoint}}-mode transition on the grace period, then retries. It requires
that only the previous deployment remains after the abort, and that the retried
child is a new resource (different \{{metadata.uid}}) whose
\{{initialSavepointPath}} is the retry's fresh savepoint. With the current
suspend behaviour the retried child is the aborted resource updated in place
(same \{{uid}}), so the test fails.
* Two existing tests (\{{verifyFailureDuringTransition}},
\{{verifyReadyTimestampResetOnAbortPreventsPrematureBlueDelete}}) asserted the
suspended child; their assertions now require that only the previous deployment
remains and keeps running.
* All Blue/Green test suites pass (133 tests).
* The change has been running on our clusters; retries after an abort start
from the serving deployment's savepoint.
A patch against \{{main}} is ready. I will open the PR once this ticket exists.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
