[
https://issues.apache.org/jira/browse/SPARK-58101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-58101:
----------------------------------
Parent: SPARK-56825
Issue Type: Sub-task (was: Bug)
> Misleading spark.submit.deployMode=cluster in
> spark-connect-server-with-spark-cluster.yaml example
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-58101
> URL: https://issues.apache.org/jira/browse/SPARK-58101
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, Kubernetes
> Affects Versions: kubernetes-operator-0.9.0
> Reporter: Yalin Dogu Sahin
> Priority: Minor
> Labels: pull-request-available
>
> [examples/spark-connect-server-with-spark-cluster.yaml|https://github.com/apache/spark-kubernetes-operator/blob/main/examples/spark-connect-server-with-spark-cluster.yaml]
> declares
> {{spark.submit.deployMode=cluster}} for a {{SparkApplication}} whose main
> class is {{org.apache.spark.sql.connect.service.SparkConnectServer}},
> attached to a standalone {{SparkCluster}} master.
> However, the operator runs the main class inside the driver pod it manages,
> so the application effectively runs in *client mode regardless of this
> setting* (verified on-cluster with operator 0.9.0 / chart 1.7.0,
> Spark 4.1.2: runtime behavior is identical with {{deployMode}} set to
> {{cluster}} and {{client}}). Upstream {{SparkSubmit}} also explicitly rejects
> cluster deploy mode for the Connect server ({{"Cluster deploy mode
> is not applicable to Spark Connect server"}}), so the example implies a
> capability that doesn't exist and a user copying it will silently get client
> mode.
> Note this is distinct from {{examples/submit-spark-connect-server.sh}}
> (SPARK-51382), which submits via the standalone REST API on port 6066 and
> bypasses the SparkSubmit-side check, so that path is not affected.
> Proposals (any of):
> # Remove the {{spark.submit.deployMode}} line from the example, or
> # Add a comment clarifying that the operator runs the Connect server in
> client mode regardless, or
> # Have the operator warn or reject when cluster deploy mode is declared for a
> Connect server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
