[
https://issues.apache.org/jira/browse/FLINK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purushottam Sinha updated FLINK-40542:
--------------------------------------
Description:
Problem
Session cluster deletion (with block-on-unmanaged-jobs, default) deletes the
cluster even when the JobManager is unreachable and running jobs can't be
enumerated — stopping unmanaged jobs without a checkpoint, with only a WARN.
Proposed fix
- getNonTerminalJobs returns Optional<Set<JobID>>; empty = indeterminate,
distinct from a present empty set.
- Indeterminate → block deletion, emit CleanupFailed, reschedule until JM is
reachable.
- Never-deployed clusters delete immediately (JM unreachable by definition,
would otherwise block forever).
- Bound this and other unbounded get() calls with client timeouts:
AbstractFlinkService#fetchSavepointInfo/#fetchCheckpointInfo/#fetchCheckpointStats/#populateStateSize,
ScalingMetricCollector#queryAggregatedMetricNames,
RestApiMetricsCollector#queryAggregatedVertexMetrics/#queryJmMetrics/#queryAggregatedTmMetrics.
Acceptance
- JM unreachable at delete time → deletion blocked, CleanupFailed emitted.
- Deletion proceeds once JM reachable and cluster empty.
- Never-deployed cluster deletes immediately.
- No FLINK-39618 deadlock regression.
- Checkpoint/savepoint/autoscaler-metric calls no longer hang indefinitely on
an unresponsive JM.
was:
Problem
When a session FlinkDeployment with block-on-unmanaged-jobs enabled (default)
is deleted while the JobManager is unreachable, the operator can't enumerate
running jobs but deletes the cluster anyway, stopping unmanaged jobs without a
checkpoint — only a WARN is logged. The safety guard fails open exactly when it
can't be evaluated.
Evidence
- SessionReconciler.getNonTerminalJobs returns Set.of() in its catch (main @
38a9f197, line 171); cleanupInternal treats an empty set as "no running jobs,
safe to delete".
- Live minikube repro on unpatched main: JM scaled to 0 + delete → Stopping
session cluster → cluster deleted (finalizer removed).
Proposed fix
- Return Optional<Set<JobID>>; empty Optional (query failed) is distinct from
present empty set (no jobs).
- On the indeterminate case, block deletion and emit a CleanupFailed event,
mirroring the block-on-session-jobs branch; reschedule so it lifts once the JM
is reachable.
- Close RestClusterClient via try-with-resources; bound the jobs request with
the Flink client timeout.
Acceptance
- With the JM unreachable at delete time, the cluster is not deleted and a
CleanupFailed event is emitted.
- Deletion proceeds once the JM is reachable and the cluster is empty.
- Does not reintroduce the FLINK-39618 deadlock (block reschedules and lifts).
> Kubernetes Operator: session cluster deleted when running jobs can't be
> queried
> -------------------------------------------------------------------------------
>
> Key: FLINK-40542
> URL: https://issues.apache.org/jira/browse/FLINK-40542
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Purushottam Sinha
> Priority: Minor
> Labels: pull-request-available
>
> Problem
> Session cluster deletion (with block-on-unmanaged-jobs, default) deletes the
> cluster even when the JobManager is unreachable and running jobs can't be
> enumerated — stopping unmanaged jobs without a checkpoint, with only a WARN.
> Proposed fix
> - getNonTerminalJobs returns Optional<Set<JobID>>; empty = indeterminate,
> distinct from a present empty set.
> - Indeterminate → block deletion, emit CleanupFailed, reschedule until JM is
> reachable.
> - Never-deployed clusters delete immediately (JM unreachable by definition,
> would otherwise block forever).
> - Bound this and other unbounded get() calls with client timeouts:
> AbstractFlinkService#fetchSavepointInfo/#fetchCheckpointInfo/#fetchCheckpointStats/#populateStateSize,
> ScalingMetricCollector#queryAggregatedMetricNames,
> RestApiMetricsCollector#queryAggregatedVertexMetrics/#queryJmMetrics/#queryAggregatedTmMetrics.
> Acceptance
> - JM unreachable at delete time → deletion blocked, CleanupFailed emitted.
> - Deletion proceeds once JM reachable and cluster empty.
> - Never-deployed cluster deletes immediately.
> - No FLINK-39618 deadlock regression.
> - Checkpoint/savepoint/autoscaler-metric calls no longer hang indefinitely on
> an unresponsive JM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
