wangzhigang1999 commented on issue #7424: URL: https://github.com/apache/kyuubi/issues/7424#issuecomment-4350011413
Thanks for filing this @oh0873. Before any fix lands I'd like to pin down a few things. **1. What Spark version are you running with Kyuubi 1.10.3?** The race window you describe lives in Spark's `KubernetesClientApplication.Client.run()`. The creation order is: 1. `preKubernetesResources` (does **not** include the driver conf ConfigMap) 2. driver Pod is created — its spec already references the not-yet-existing `*-driver-conf-map` as a volume 3. ownerRefs added back to pre-resources 4. driver ConfigMap (and other "after-pod" resources) are created, with `ownerRef → Pod` Sources: [Spark v3.5.3](https://github.com/apache/spark/blob/v3.5.3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L137-L181), [Spark master](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L137-L181). If the spark-submit JVM (the Kyuubi server) is killed between step 2 and step 4 (SIGKILL / OOM / host crash / restart), the result is: - Pod exists in apiserver, referencing a ConfigMap that will never be created - Pod has no `ownerReferences` → K8s GC won't reclaim it - The CM was never created → its ownerRef-cascade can't help either - Kyuubi's `appInfoStore` is in-memory, so after restart Kyuubi has no record of this pod either Upstream is aware but unfixed, and the same code path is present in master: - [SPARK-38079](https://issues.apache.org/jira/browse/SPARK-38079) — Not waiting for configmap before starting driver - [SPARK-44050](https://issues.apache.org/jira/browse/SPARK-44050) — Unable to Mount ConfigMap in Driver Pod (~5% submission impact reported on 3.3.1) The probability of hitting the window is heavily version-dependent in practice — older Spark releases had longer step-2→step-4 latency, while 3.5+ with server-side apply usually closes it in tens of milliseconds. Hence the version question. **2. Is this scoped to `FailedMount`, or to orphan driver pods in general?** The same crash window can produce orphans with other reasons (`ImagePullBackOff`, `FailedScheduling`, terminal pods no longer tracked by any session). The current title and PR target `FailedMount` specifically. It would help to know whether this issue is intentionally narrow, or whether the broader orphan-pod class is in scope. **3. How does this interact with `kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind`?** Kyuubi already has [this config](https://github.com/apache/kyuubi/blob/master/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala#L1387) (since 1.8.1, default `NONE`) which intentionally keeps terminated driver pods around for post-mortem investigation — pod events, kubelet logs, the missing CM name. `FailedMount` is exactly the kind of state operators usually want to preserve. Any cleanup mechanism added on top needs a clear answer for what happens to users who rely on `NONE`. **4. Verifying that the proposed approach actually targets this scenario.** The orphans described here arise from Kyuubi server crash/restart, after which `appInfoStore` (in-memory `ConcurrentHashMap`) is empty. A periodic checker iterating over `appInfoStore` would not see them. Could you walk through how the proposed fix is expected to clean up the specific pods this issue is about? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
