Dongjoon,
Thanks for fixing this!
https://github.com/apache/spark/pull/55355
Enrico
Am 15.04.26 um 18:22 schrieb Dongjoon Hyun:
`spark-app-selector` works fine in our CIs and my environment like the
following. Could you double-check your environment, Enrico?
$ bin/docker-image-tool.sh -r docker.io/apache -t 4.2.0-preview4 build
$ bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
-c spark.kubernetes.container.image=apache/spark:4.2.0-preview4 \
-c spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 300000
$ kubectl get pods --show-labels
NAME READY STATUS
RESTARTS AGE LABELS
org-apache-spark-examples-sparkpi-8fea449d91f10411-driver 1/1 Running 0
6s
spark-app-name=org-apache-spark-examples-sparkpi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-role=driver,spark-version=4.2.0-preview4
spark-pi-e33c809d91f10f25-exec-1 1/1 Running 0
3s
spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=1,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
spark-pi-e33c809d91f10f25-exec-2 1/1 Running 0
3s
spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=2,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
Dongjoon.
On 2026/04/14 09:04:44 Enrico Minack wrote:
Hi Spark-devs,
Looks like 4.2.0-preview4 is affected by regression SPARK-56238.
Building a docker image from 4.2.0-preview4 and running a Spark job on
Kubernetes fails due to a Spark app id mismatch in the selector.
Setting an app id explicitly fixes this mismatch: --conf spark.app.id=abc
Build Spark binaries and Docker image, load image into minikube:
cd spark-4.2.0-preview4-bin-hadoop3
./bin/docker-image-tool.sh -t "latest" build
docker save spark:latest -o spark.tar
minikube image load spark.tar
Run Spark Shell:
echo"spark.range(10).mapPartitions { it => Thread.sleep(60000); it }.collect" |
./bin/spark-shell --master k8s://https://192.168.49.2:8443--conf
spark.kubernetes.container.image=spark:latest --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf
spark.kubernetes.executor.missingPodDetectDelta=1000
The driver says:
Spark context available as'sc' (master = k8s://https://192.168.49.2:8443, app
id =
spark-c0f802279d8146c29b1ef3467b467590).
The executor pods say:
kubectl describe"pod/$(kubectl get pods | grep exec | grep Running | head -n 1 | cut -d "
" -f 1)" | grep selec
spark-app-selector=spark-6c8d69be4f00410b884fd6e6417b872a
this mismatch causes:
26/03/26 11:40:29 ERROR dispatcher-CoarseGrainedScheduler
org.apache.spark.scheduler.TaskSchedulerImpl: Lost executor 1 on 10.244.0.55:
The executor with ID 1 (registered at 1774521602640 ms) was not found in the
cluster at the polling time (1774521629051 ms) which is after the accepted
detect delta time (1000 ms) configured by
`spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been
deleted but the driver missed the deletion event. Markingthis executor as
failed.
Either the app id mismatch should be fixed, or the app id should be
mandatory in this setup.
Cheers,
Enrico
Am 10.04.26 um 03:06 schrieb Hyukjin Kwon:
Hi all,
To enable wide-scale community testing of the upcoming Spark 4.2.0
release,
the Apache Spark community has posted a Spark 4.2.0-preview4 release
<https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4>.
This preview is not a stable release in terms of either API or
functionality,
but it is meant to give the community early access to try the code
that will
become Spark 4.2.0. If you would like to test the release, please
download
<https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview3> it,
and send feedback using either mailing lists
<https://spark.apache.org/community.html> or JIRA
<https://issues.apache.org/jira/browse/SPARK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>.
The documentation is available at the link
<https://spark.apache.org/docs/4.2.0-preview3/>.
We’d like to thank our contributors and users for their contributions
and early feedback to this release. This release would not have been
possible without you.
To download Spark 4.2.0-preview4, head over to the download page:
https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/
<https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/>
To view the documentation:
https://spark.apache.org/docs/preview/
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
