Peter Toth created SPARK-59578:
----------------------------------
Summary: Clarify the Kueue `integrations.externalFrameworks`
recommendation in `docs/operations.md`
Key: SPARK-59578
URL: https://issues.apache.org/jira/browse/SPARK-59578
Project: Spark
Issue Type: Sub-task
Components: Kubernetes
Affects Versions: kubernetes-operator-1.1.0
Reporter: Peter Toth
docs/operations.md:40-45 lists Kueue's integrations.externalFrameworks
registration of SparkApplication.v1.spark.apache.org and
SparkCluster.v1.spark.apache.org as something "required" whenever
operatorRbac.kueue.enabled is set. That overstates it, and CI does not do it:
the kueue jobs install the stock manifests.yaml, which registers neither.
Analysis by Dongjoon Hyun on
https://github.com/apache/spark-kubernetes-operator/pull/829#discussion_r4027226533,
checked against Kueue v0.19.4:
- integrations.externalFrameworks is only read on the job framework paths -
pkg/controller/jobframework/reconciler.go:919,
pkg/controller/jobframework/defaults.go:88 and :107, and
pkg/controller/jobs/pod/pod_webhook.go:269, where it only builds a warning
string.
- The Workload controller, the scheduler and the Workload webhook never look at
the owner kind. The only owner check in
pkg/controller/core/workload_controller.go is isOrphanedWorkload, which fires
only on empty ownerReferences. The operator always sets a controller reference,
so a Workload it creates is admitted without the registration.
So the registration is not a hard prerequisite of operatorRbac.kueue.enabled.
It does matter in one concrete case, and that is what the doc should say
instead:
- defaultLocalQueueApplies (pkg/controller/jobframework/defaults.go:80-88) has
no feature gate, unlike ApplyDefaultWorkloadPriorityClass immediately below it.
It returns true when the namespace has a LocalQueue named "default", the object
carries no queue-name, and the owner is not a kind Kueue manages.
- So in a namespace with a default LocalQueue, an unregistered SparkApplication
means Kueue adds queue-name: default to the driver pod the operator creates,
queueing that pod on its own.
{code}
func defaultLocalQueueApplies(jobObj client.Object, defaultQueueExist
func(string) bool) bool {
if !defaultQueueExist(jobObj.GetNamespace()) {
return false
}
if QueueNameForObject(jobObj) != "" {
return false
}
// Do not default the queue-name for a job whose owner is already
managed by Kueue
return !IsOwnerManagedByKueueForObject(jobObj)
}
{code}
Proposed change: reword docs/operations.md:40-45 to present the registration as
recommended rather than required, and to name the default-LocalQueue
pod-queueing case as the reason. No CI change - PR #829 leaves the stock
install in place deliberately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
