Wenjun Ruan created SPARK-59725:
-----------------------------------
Summary: Create Volcano PodGroup with driver owner reference after
driver pod creation
Key: SPARK-59725
URL: https://issues.apache.org/jira/browse/SPARK-59725
Project: Spark
Issue Type: Bug
Components: Kubernetes, Scheduler
Affects Versions: 4.3.0
Reporter: Wenjun Ruan
h1. Description
When submitting a Spark application with Volcano, Spark currently performs the
following operations:1. Create the PodGroup without an owner reference.
2. Create the driver pod.
3. Update the PodGroup with an owner reference to the driver pod.
This requires two writes to the PodGroup. If the submission client fails or
exits before the owner reference is attached, the PodGroup may remain without
an owner reference, preventing Kubernetes garbage collection from cleaning it
up when the driver pod is deleted.
h1. Proposed change
Create the driver pod first, then create the PodGroup with the driver pod's
owner reference already populated. This can use the existing post-pod resource
creation mechanism by moving the PodGroup from
`getAdditionalPreKubernetesResources()` to `getAdditionalKubernetesResources()`
in `VolcanoFeatureStep`.
h1. Expected benefits
- Eliminate the additional PodGroup update used to attach the owner reference.
- Eliminate the window in which a newly created PodGroup has no driver owner
reference.
- Preserve the existing cleanup behavior when post-pod resource creation fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
