roryqi commented on PR #13700:
URL: https://github.com/apache/gravitino/pull/13700#issuecomment-6187058021
Thanks for the work. I checked the Spark deploy-mode behavior and the
Kubernetes compatibility boundary, and I think two points should be clarified
before this executor is registered and shipped.
1. **Please reject client mode instead of silently running it in cluster
mode.**
`SparkApplicationUtils` removes `spark.submit.deployMode` unconditionally
and does not set `spec.deploymentMode`, so the operator uses its default
`ClusterMode`. Consequently, a template that explicitly requests
`spark.submit.deployMode=client` is silently executed in cluster mode. Apache
Spark Kubernetes Operator 0.9.0 does not support client mode and explicitly
fails a `ClientMode` SparkApplication, so I think the executor should accept an
unset/`cluster` value but fail submission with a clear
`IllegalArgumentException` for `client`. The executor documentation should also
state that only Spark cluster deploy mode is supported.
Operator behavior:
https://github.com/apache/spark-kubernetes-operator/blob/0.9.0/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppValidateStep.java#L53-L60
2. **Please document the job-cluster compatibility matrix separately from
the Gravitino deployment requirements.**
Operator 0.9.0 officially lists Kubernetes 1.34, 1.35, and 1.36, and
Spark 3.5/4.0/4.1 (plus testing with a 4.2 preview). This is narrower than the
Gravitino Helm chart's Kubernetes 1.29+ prerequisite. Users need to know that
the Gravitino host cluster and the Spark job cluster can have different
requirements; otherwise a same-cluster installation on Kubernetes 1.29-1.33
looks supported when the selected operator version does not officially support
it. Kubernetes 1.37 is also outside the published 0.9.0 matrix.
The bundled Fabric8 client is 7.2.0, whose release added Kubernetes 1.33
model support. Since this implementation uses stable Pod APIs and a generic CRD
client, I do not see a definite protocol incompatibility with Kubernetes
1.34-1.36, but the mock-server tests and the OrbStack test without a stated
Kubernetes version do not establish that compatibility. Please publish the
Operator/Kubernetes/Spark version matrix and run at least one real-cluster
smoke test on a supported Kubernetes version before shipping.
Operator 0.9.0 matrix:
https://github.com/apache/spark-kubernetes-operator/releases/tag/0.9.0
Fabric8 7.2.0 release:
https://github.com/fabric8io/kubernetes-client/releases/tag/v7.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
