`spark-app-selector` works fine in our CIs and my environment like the
following. Could you double-check your environment, Enrico?
$ bin/docker-image-tool.sh -r docker.io/apache -t 4.2.0-preview4 build
$ bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
-c spark.kubernetes.container.image=apache/spark:4.2.0-preview4 \
-c spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 300000
$ kubectl get pods --show-labels
NAME READY STATUS
RESTARTS AGE LABELS
org-apache-spark-examples-sparkpi-8fea449d91f10411-driver 1/1 Running 0
6s
spark-app-name=org-apache-spark-examples-sparkpi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-role=driver,spark-version=4.2.0-preview4
spark-pi-e33c809d91f10f25-exec-1 1/1 Running 0
3s
spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=1,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
spark-pi-e33c809d91f10f25-exec-2 1/1 Running 0
3s
spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=2,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
Dongjoon.
On 2026/04/14 09:04:44 Enrico Minack wrote:
> Hi Spark-devs,
>
> Looks like 4.2.0-preview4 is affected by regression SPARK-56238.
>
> Building a docker image from 4.2.0-preview4 and running a Spark job on
> Kubernetes fails due to a Spark app id mismatch in the selector.
>
> Setting an app id explicitly fixes this mismatch: --conf spark.app.id=abc
>
> Build Spark binaries and Docker image, load image into minikube:
>
> cd spark-4.2.0-preview4-bin-hadoop3
> ./bin/docker-image-tool.sh -t "latest" build
> docker save spark:latest -o spark.tar
> minikube image load spark.tar
>
> Run Spark Shell:
>
> echo"spark.range(10).mapPartitions { it => Thread.sleep(60000); it }.collect"
> | ./bin/spark-shell --master k8s://https://192.168.49.2:8443--conf
> spark.kubernetes.container.image=spark:latest --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf
> spark.kubernetes.executor.missingPodDetectDelta=1000
>
> The driver says:
>
> Spark context available as'sc' (master = k8s://https://192.168.49.2:8443, app
> id =
> spark-c0f802279d8146c29b1ef3467b467590).
>
> The executor pods say:
>
> kubectl describe"pod/$(kubectl get pods | grep exec | grep Running | head -n
> 1 | cut -d " " -f 1)" | grep selec
> spark-app-selector=spark-6c8d69be4f00410b884fd6e6417b872a
>
> this mismatch causes:
>
> 26/03/26 11:40:29 ERROR dispatcher-CoarseGrainedScheduler
> org.apache.spark.scheduler.TaskSchedulerImpl: Lost executor 1 on 10.244.0.55:
> The executor with ID 1 (registered at 1774521602640 ms) was not found in the
> cluster at the polling time (1774521629051 ms) which is after the accepted
> detect delta time (1000 ms) configured by
> `spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been
> deleted but the driver missed the deletion event. Markingthis executor as
> failed.
>
>
> Either the app id mismatch should be fixed, or the app id should be
> mandatory in this setup.
>
> Cheers,
> Enrico
>
>
> Am 10.04.26 um 03:06 schrieb Hyukjin Kwon:
> > Hi all,
> >
> > To enable wide-scale community testing of the upcoming Spark 4.2.0
> > release,
> > the Apache Spark community has posted a Spark 4.2.0-preview4 release
> > <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4>.
> > This preview is not a stable release in terms of either API or
> > functionality,
> > but it is meant to give the community early access to try the code
> > that will
> > become Spark 4.2.0. If you would like to test the release, please
> > download
> > <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview3> it,
> > and send feedback using either mailing lists
> > <https://spark.apache.org/community.html> or JIRA
> > <https://issues.apache.org/jira/browse/SPARK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>.
> >
> > The documentation is available at the link
> > <https://spark.apache.org/docs/4.2.0-preview3/>.
> >
> > We’d like to thank our contributors and users for their contributions
> > and early feedback to this release. This release would not have been
> > possible without you.
> >
> > To download Spark 4.2.0-preview4, head over to the download page:
> > https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/
> > <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/>
> >
> > To view the documentation:
> > https://spark.apache.org/docs/preview/
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
