[
https://issues.apache.org/jira/browse/SPARK-59725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenjun Ruan updated SPARK-59725:
--------------------------------
Summary: Bind Volcano PodGroup with driver when podgroup creation (was:
Create Volcano PodGroup with driver owner reference after driver pod creation)
> Bind Volcano PodGroup with driver when podgroup creation
> --------------------------------------------------------
>
> Key: SPARK-59725
> URL: https://issues.apache.org/jira/browse/SPARK-59725
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Scheduler
> Affects Versions: 4.3.0
> Reporter: Wenjun Ruan
> Priority: Minor
> Labels: pull-request-available
>
> h1. Description
>
> When submitting a Spark application with Volcano, Spark currently performs
> the following operations:
> # Create the PodGroup without an owner reference.
> # Create the driver pod
> # Update the PodGroup with an owner reference to the driver pod.
> This requires two writes to the PodGroup. If the submission client fails or
> exits before the owner reference is attached, the PodGroup may remain without
> an owner reference, preventing Kubernetes garbage collection from cleaning it
> up when the driver pod is deleted.
>
> h1. Proposed change
> Create the driver pod first, then create the PodGroup with the driver pod's
> owner reference already populated. This can use the existing post-pod
> resource creation mechanism by moving the PodGroup from
> `getAdditionalPreKubernetesResources()` to
> `getAdditionalKubernetesResources()` in `VolcanoFeatureStep`.
>
> h1. Expected benefits
> * Eliminate the additional PodGroup update used to attach the owner
> reference.
> * Eliminate the window in which a newly created PodGroup has no driver owner
> reference.
> * Preserve the existing cleanup behavior when post-pod resource creation
> fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
