[
https://issues.apache.org/jira/browse/SPARK-59518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-59518:
-----------------------------
Description:
{{ExecutorKubernetesCredentialsFeatureStep}} treats
{{spark.kubernetes.authenticate.executor.serviceAccountName=""}} as set: an
empty value shadows a non-empty
{{spark.kubernetes.authenticate.driver.serviceAccountName}} and is written into
both {{serviceAccount}} and {{serviceAccountName}} on the executor pod, leaving
it on the namespace default.
The step reads the same value the same way when it reports an account that did
not apply: with an empty executor value and a template that names an account,
the warning prints the empty value, {{Not applying
spark.kubernetes.authenticate.executor.serviceAccountName= to the executor pod,
because its pod template already names ...}}.
The configuration's own doc says the driver account is the fallback "if this
parameter is not setup", and the template side of SPARK-58910 counts an empty
value as unset, matching Kubernetes' {{SetDefaults_PodSpec}}. So the same
notion of "unset" is read two ways in one step.
Aligning both paths means {{.filter(_.nonEmpty)}} on the two {{lazy val}}s in
that step, which fixes the write and the warning together. It is a small
user-visible change: an empty executor value would fall back to the driver
account instead of writing an empty one, so it is filed separately from
SPARK-58910 rather than folded into that fix.
was:
{{ExecutorKubernetesCredentialsFeatureStep}} treats
{{spark.kubernetes.authenticate.executor.serviceAccountName=""}} as set: an
empty value shadows a non-empty
{{spark.kubernetes.authenticate.driver.serviceAccountName}} and is written into
both {{serviceAccount}} and {{serviceAccountName}} on the executor pod, leaving
it on the namespace default.
The configuration's own doc says the driver account is the fallback "if this
parameter is not setup", and the template side of SPARK-58910 counts an empty
value as unset, matching Kubernetes' {{SetDefaults_PodSpec}}. So the same
notion of "unset" is read two ways in one step.
Aligning the write path means {{.filter(_.nonEmpty)}} on both {{lazy val}}s in
that step. It is a small user-visible change: an empty executor value would
fall back to the driver account instead of writing an empty one, so it is filed
separately from SPARK-58910 rather than folded into a fix headed for the
maintenance branches.
> An empty spark.kubernetes.authenticate.executor.serviceAccountName is written
> to the executor pod instead of falling back to the driver account
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59518
> URL: https://issues.apache.org/jira/browse/SPARK-59518
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> {{ExecutorKubernetesCredentialsFeatureStep}} treats
> {{spark.kubernetes.authenticate.executor.serviceAccountName=""}} as set: an
> empty value shadows a non-empty
> {{spark.kubernetes.authenticate.driver.serviceAccountName}} and is written
> into both {{serviceAccount}} and {{serviceAccountName}} on the executor pod,
> leaving it on the namespace default.
> The step reads the same value the same way when it reports an account that
> did not apply: with an empty executor value and a template that names an
> account, the warning prints the empty value, {{Not applying
> spark.kubernetes.authenticate.executor.serviceAccountName= to the executor
> pod, because its pod template already names ...}}.
> The configuration's own doc says the driver account is the fallback "if this
> parameter is not setup", and the template side of SPARK-58910 counts an empty
> value as unset, matching Kubernetes' {{SetDefaults_PodSpec}}. So the same
> notion of "unset" is read two ways in one step.
> Aligning both paths means {{.filter(_.nonEmpty)}} on the two {{lazy val}}s in
> that step, which fixes the write and the warning together. It is a small
> user-visible change: an empty executor value would fall back to the driver
> account instead of writing an empty one, so it is filed separately from
> SPARK-58910 rather than folded into that fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
