[
https://issues.apache.org/jira/browse/SPARK-59758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-59758:
----------------------------------
Summary: Release the Kueue `Workload` of a terminated `SparkApplication`
regardless of the queue label (was: Release the Kueue `Workload` of suspended
`Spark(Application|Cluster)` regardless of the queue label)
> Release the Kueue `Workload` of a terminated `SparkApplication` regardless of
> the queue label
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-59758
> URL: https://issues.apache.org/jira/browse/SPARK-59758
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.1.0
> Reporter: Peter Toth
> Priority: Major
>
> SPARK-59754 removed the `kueue.x-k8s.io/queue-name` label check from the two
> suspension paths, so a `Workload` created before the label was removed is
> released. Two release paths in `AppCleanUpStep` still gate on the label and
> keep the same leak:
> * `AppCleanUpStep` releases the `Workload` of a terminating or restarting
> attempt only `if (KueueWorkloadFactory.hasQueueName(application))`.
> * `finishKueueWorkloadOfRetainedApp` records the Kueue `Finished` condition
> only under the same check.
> Measured on an application reaching `Succeeded`, with a mocked
> `KueueWorkloadUtils`:
> {noformat}
> labeled, resourceRetainPolicy=Never -> releaseWorkload called
> unlabeled, resourceRetainPolicy=Never -> neither called
> labeled, resourceRetainPolicy=Always -> finishWorkload called
> unlabeled, resourceRetainPolicy=Always -> neither called
> {noformat}
> So an application whose queue label was removed after its `Workload` was
> admitted keeps that quota until its custom resource is deleted, which with
> the default `ttlAfterStopMillis` of -1 may be never.
> After SPARK-59754, dropping both checks is safe:
> `KueueWorkloadUtils.deleteWorkloadOf` already treats a `403 Forbidden` as no
> `Workload` when the owner has no queue label and the `Workload` informer is
> disabled, so an operator that was never granted access to `Workload`s is
> unaffected. `releaseWorkload` swallows every other failure, and
> `finishWorkload` falls back to `releaseWorkload`.
> The two admission gates, `ClusterInitStep.holdForKueueAdmission` and the
> equivalent in `AppInitStep`, must keep the label check: an unlabeled resource
> has no admission to request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
