Peter Toth created SPARK-59606:
----------------------------------
Summary: Map `SparkCluster` pods by
`spark.operator/spark-cluster-name` in `SparkClusterReconciler`
Key: SPARK-59606
URL: https://issues.apache.org/jira/browse/SPARK-59606
Project: Spark
Issue Type: Sub-task
Components: Kubernetes
Affects Versions: kubernetes-operator-1.1.0
Reporter: Peter Toth
SparkClusterReconciler.prepareEventSources registers the Pod informer with
basicLabelSecondaryToPrimaryMapper(LABEL_SPARK_APPLICATION_NAME), i.e. it maps
a Pod back to its primary SparkCluster by reading the label
spark.operator/spark-app-name.
SparkCluster pods never carry that label. SparkClusterResourceSpec adds only
spark.operator/spark-role, spark.operator/spark-cluster-name and
spark.operator/spark-version to the master and worker pod templates (lines
234-237 and 316-319), and Utils.sparkClusterResourceLabels adds
spark.operator/spark-cluster-name plus commonManagedResourceLabels, which is
only spark.operator/name. LABEL_SPARK_APPLICATION_NAME is set exclusively by
Utils.sparkAppResourceLabels, for SparkApplication resources.
So Utils.basicLabelSecondaryToPrimaryMapper takes its `name == null` branch and
returns Set.of() for every SparkCluster pod. The pods still match the
informer's commonResourceLabelsStr() selector and are cached, but no primary
ResourceID is produced, so a Pod event never triggers a SparkCluster
reconciliation. An owner-reference mapper would not help either: cluster pods
are owned by their StatefulSet, not by the SparkCluster, which is presumably
why a label mapper is used.
Effect: a master or worker pod dying, becoming ready, or being evicted is not
noticed until the next periodic reconcile,
spark.kubernetes.operator.reconciler.intervalSeconds, 120 seconds by default.
SparkApplication is unaffected; SparkAppReconciler uses the same mapper with
the label its pods actually carry.
The fix is to use LABEL_SPARK_CLUSTER_NAME in
SparkClusterReconciler.prepareEventSources, which is what PR #836 does for the
new Kueue Workload informer three lines below.
Worth adding a test that asserts the mapper resolves a realistic cluster pod to
its SparkCluster. The existing SparkClusterReconcilerTest only asserts the
number of event sources and their resource classes, so it does not catch a
wrong label: swapping the label in the Workload source leaves all 11 tests
green.
Found while reviewing
https://github.com/apache/spark-kubernetes-operator/pull/836
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
