[
https://issues.apache.org/jira/browse/SPARK-59008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-59008.
-----------------------------------
Fix Version/s: 4.1.4
4.2.1
4.3.0
Resolution: Fixed
Issue resolved by pull request 58291
[https://github.com/apache/spark/pull/58291]
> NullPointerException in ExecutorPodsLifecycleManager when pod is deleted
> between get() calls
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-59008
> URL: https://issues.apache.org/jira/browse/SPARK-59008
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 4.1.1
> Reporter: Venkata krishnan Sowrirajan
> Assignee: Venkata krishnan Sowrirajan
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.4, 4.2.1, 4.3.0
>
>
> ExecutorPodsLifecycleManager.removeExecutorFromK8s calls "podToDelete.get()"
> twice:
> {code:java}
> if (podToDelete.get() != null &&
> podToDelete.get.getMetadata.getDeletionTimestamp == null) {
> {code}
> podToDelete is a lazy PodResource handle, so each get() issues a separate
> API-server call. If the pod is deleted between the two calls, the second
> returns null and dereferencing getMetadata throws:
> {code:java}
> java.lang.NullPointerException: Cannot invoke
> "io.fabric8.kubernetes.api.model.Pod.getMetadata()"
> because the return value of
> "io.fabric8.kubernetes.client.dsl.PodResource.get()" is null
> at
> ExecutorPodsLifecycleManager.$anonfun$removeExecutorFromK8s$1(ExecutorPodsLifecycleManager.scala:222)
> at
> ExecutorPodsLifecycleManager.onFinalNonDeletedState(ExecutorPodsLifecycleManager.scala:194)
> at
> ExecutorPodsLifecycleManager.onNewSnapshots(ExecutorPodsLifecycleManager.scala:98)
> {code}
> Observed on a driver managing ~1000 executors during heavy pod churn. The
> exception aborts the K8s-side deletion for that executor (Spark-side
> accounting in removeExecutorFromSpark has already completed), leaving the pod
> to be reaped on a later resync and logging a spurious uncaught-exception
> error.
> Fix: fetch the pod once and test that single value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
