HwangDongJun opened a new pull request, #7750: URL: https://github.com/apache/kyuubi/pull/7750
### Why are the changes needed? When submitting a Spark batch job via the Batch REST API with `spark.master=k8s://...`, Kyuubi does not validate that `spark.submit.deployMode` is `cluster`. If `deployMode` is `client` (Spark's own default when unset), Kyuubi still proceeds to submit the job, and Spark runs the driver as a subprocess of the Kyuubi server itself instead of creating a separate Kubernetes driver pod. This breaks batch status tracking (which relies on finding a driver pod tagged with `kyuubi-unique-tag`) and produces confusing errors that give no indication of the actual root cause. Root cause is two compounding issues: 1. `BatchJobSubmission` never calls `builder.validateConf()` at all, so the existing `Validator.validateConf()` safety check (currently validating `spark.kubernetes.executor.podNamePrefix` length) has never been exercised for any batch job. 2. Even when invoked, `Validator` has no check for the `spark.master` / `deployMode` combination. This adds an override of `validateConf()` in `SparkBatchProcessBuilder` that fails fast with a clear error when `spark.master` is `k8s://` and `deployMode` is not `cluster`, and wires up the missing `validateConf()` call in `BatchJobSubmission`. See #7749 for full reproduction details. ### How was this patch tested? Added 4 new unit tests to `SparkBatchProcessBuilderSuite`: - reject k8s batch submission when `deployMode` is `client` - reject k8s batch submission when `deployMode` is unset - allow k8s batch submission when `deployMode` is `cluster` - do not reject non-k8s masters (`local[*]`, `yarn` client) regardless of `deployMode` Ran the full `kyuubi-server` test suite locally (`mvn test -pl kyuubi-server -am -P spark-3.5`) — all 6 tests in `SparkBatchProcessBuilderSuite` pass, including the 2 pre-existing tests (no regression). Also verified via a fork-internal CI dry-run (base/head both in my own fork) that the full CI matrix passes with this change: all 33 checks green, including `Kyuubi and Spark Test` across Java 17/21/25 × Spark 3.5/4.0/4.1/4.2, `Kyuubi Server On Kubernetes Integration Test`, `Spark Engine On Kubernetes Integration Test`, and `Style check`. ### Was this patch assisted by generative AI tooling? Assisted-by: claude-sonnet-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
