jerryshao opened a new pull request, #13948: URL: https://github.com/apache/gravitino/pull/13948
### What changes were proposed in this pull request? The `k8s` job executor rejected Spark job templates with environment variables. It now passes them to the Spark driver and executors. - Each environment variable `X` of the job template becomes the Spark configurations `spark.kubernetes.driverEnv.X` and `spark.executorEnv.X`. - They override the default Spark configurations of the executor (`gravitino.jobExecutor.k8s.spark.conf.*`), and the Spark configurations of the job template override them, so a template can still set a different value for the driver or the executors. - A name that Kubernetes doesn't accept in all its versions, which must match `[-._a-zA-Z][-._a-zA-Z0-9]*`, is rejected when the job is submitted, instead of failing the job later when the operator creates the driver pod. The error message names the variable, never its value. The values are readable in the SparkApplication and in its pods. A sensitive value can come from a Secret created beforehand, referenced in the Spark configurations with `spark.kubernetes.driver.secretKeyRef.<name>` and `spark.kubernetes.executor.secretKeyRef.<name>`, which needs no change in Gravitino. #13618 documents both. ### Why are the changes needed? A job template with environment variables runs on the `local` job executor but was rejected by the `k8s` one. Fix: #13617 ### Does this PR introduce _any_ user-facing change? Yes. Spark job templates with environment variables run on the `k8s` job executor, and an invalid variable name is rejected with an error. No new configuration. ### How was this patch tested? - Unit tests for the mapping, the precedence between the executor defaults, the environment variables and the template configurations, and the rejected names. - Manually, with a Gravitino server started from the distribution package of this branch against a local Kubernetes cluster (OrbStack) with spark-kubernetes-operator 0.9.0, through the REST API: a Spark Pi template with environment variables, one of them from a placeholder, succeeded. The values were checked in the pod specs and in the processes of the driver and the executor, including a name with a dot, a server default overridden by the template, a template configuration overriding a variable for the executors only, and a variable from a Secret through `secretKeyRef` in the template configurations. A template with the variable name `MY VAR` was rejected with a 400. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
