DSurfer opened a new issue, #539:
URL: https://github.com/apache/spark-kubernetes-operator/issues/539
I was able to start cluster but everytime i add any parameters to sparkConf
nothing happens
`---
apiVersion: spark.apache.org/v1
kind: SparkCluster
metadata:
name: cluster
namespace: spark-operator
spec:
runtimeVersions:
sparkVersion: "4.1.1"
clusterTolerations:
instanceConfig:
initWorkers: 1
minWorkers: 1
maxWorkers: 1
masterSpec:
statefulSetSpec:
template:
spec:
nodeSelector:
spark: dedicated
tolerations:
- key: spark
operator: Equal
value: dedicated
effect: NoSchedule
workerSpec:
statefulSetSpec:
template:
spec:
nodeSelector:
spark: dedicated
tolerations:
- key: spark
operator: Equal
value: dedicated
effect: NoSchedule
sparkConf:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}"
spark.master.ui.title: "Test Spark Cluster"
spark.master.rest.enabled: "true"
spark.master.rest.host: "0.0.0.0"
spark.ui.reverseProxy: "true"`
Added Conf but operator is not syncing
` spark.driver.host: "cluster-master-svc.spark-operator.svc.cluster.local"
spark.driver.bindAddress: "0.0.0.0"`
> Starting Operator...
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Configuring operator with
50 reconciliation threads.
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Adding Operator
JosdkMetrics to metrics system.
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Configuring operator with
50 reconciliation threads.
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Adding Operator
JosdkMetrics to metrics system.
26/03/05 13:21:08 INFO i.j.o.a.c.BaseConfigurationService Created
configuration for reconciler
org.apache.spark.k8s.operator.reconciler.SparkAppReconciler with name
sparkappreconciler
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Initializing with watched
namespaces [spark-operator]
26/03/05 13:21:08 INFO o.a.s.k.o.c.SparkOperatorConf Reconciler retry
policy is configured with unlimited max attempts
26/03/05 13:21:08 INFO i.j.o.Operator Registered reconciler:
'sparkappreconciler' for resource: 'class
org.apache.spark.k8s.operator.SparkApplication' for namespace(s):
[spark-operator]
26/03/05 13:21:08 INFO i.j.o.a.c.BaseConfigurationService Created
configuration for reconciler
org.apache.spark.k8s.operator.reconciler.SparkClusterReconciler with name
sparkclusterreconciler
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Initializing with watched
namespaces [spark-operator]
26/03/05 13:21:08 INFO o.a.s.k.o.c.SparkOperatorConf Reconciler retry
policy is configured with unlimited max attempts
26/03/05 13:21:08 INFO i.j.o.Operator Registered reconciler:
'sparkclusterreconciler' for resource: 'class
org.apache.spark.k8s.operator.SparkCluster' for namespace(s): [spark-operator]
26/03/05 13:21:08 INFO o.a.s.k.o.SparkOperator Starting conf monitor in
namespace: spark-operator, with selector:
app.kubernetes.io/name=spark-kubernetes-operator,app.kubernetes.io/component=operator-dynamic-config-overrides
26/03/05 13:21:08 INFO i.j.o.a.c.BaseConfigurationService Created
configuration for reconciler
org.apache.spark.k8s.operator.config.SparkOperatorConfigMapReconciler with name
sparkoperatorconfigmapreconciler
26/03/05 13:21:08 INFO i.j.o.Operator Registered reconciler:
'sparkoperatorconfigmapreconciler' for resource: 'class
io.fabric8.kubernetes.api.model.ConfigMap' for namespace(s): [spark-operator]
26/03/05 13:21:08 INFO i.j.o.Operator Operator SDK 5.2.2 (commit: c6416e0)
built on 2026-01-07T13:01:04.000+0000 starting...
26/03/05 13:21:08 INFO i.j.o.Operator Client version: 7.4.0
26/03/05 13:21:08 INFO i.j.o.p.Controller Starting 'sparkappreconciler'
controller for reconciler:
org.apache.spark.k8s.operator.reconciler.SparkAppReconciler, resource:
org.apache.spark.k8s.operator.SparkApplication
26/03/05 13:21:08 INFO i.j.o.p.Controller Starting
'sparkclusterreconciler' controller for reconciler:
org.apache.spark.k8s.operator.reconciler.SparkClusterReconciler, resource:
org.apache.spark.k8s.operator.SparkCluster
26/03/05 13:21:13 INFO i.j.o.p.Controller 'sparkappreconciler' controller
started
26/03/05 13:21:13 INFO i.j.o.p.Controller 'sparkclusterreconciler'
controller started
26/03/05 13:21:13 INFO i.j.o.Operator Operator SDK 5.2.2 (commit: c6416e0)
built on 2026-01-07T13:01:04.000+0000 starting...
26/03/05 13:21:13 INFO i.j.o.Operator Client version: 7.4.0
26/03/05 13:21:13 INFO i.j.o.p.Controller Starting
'sparkoperatorconfigmapreconciler' controller for reconciler:
org.apache.spark.k8s.operator.config.SparkOperatorConfigMapReconciler,
resource: io.fabric8.kubernetes.api.model.ConfigMap
26/03/05 13:21:13 INFO i.j.o.p.Controller
'sparkoperatorconfigmapreconciler' controller started
26/03/05 13:21:13 INFO o.a.s.k.o.p.ProbeService Probe service started at
19091
26/03/05 13:21:13 INFO spark-kubernetes-operator-dynamic-configuration
spark-operator o.a.s.k.o.SparkOperator No watched namespace change detected
26/03/05 13:21:13 INFO o.a.s.k.o.m.MetricsSystem sinkPropertiesMap: {}
26/03/05 13:21:13 INFO o.a.s.k.o.m.PrometheusPullModelHandler
PrometheusPullModelHandler started
26/03/05 13:21:13 INFO o.a.s.k.o.m.MetricsService Starting Metrics Service
for Prometheus ...
values for operator
`values:
operatorDeployment:
replicas: 1
strategy:
type: Recreate
# workloadResources:
# namespaces:
# create: true
# # When enabled, value for conf property
`spark.operator.watched.namespaces` would be set to
# # the values provided in {workloadResources.namespaces.data} field
as well
# overrideWatchedNamespaces: true
# data:
# - spark
operatorConfiguration:
spark-operator.properties: |+
spark.kubernetes.operator.dynamicConfig.enabled=true
spark.kubernetes.operator.reconciler.intervalSeconds=5
dynamicConfig:
# Enable this for hot properties loading.
enable: true
# Enable this to create a config map for hot property loading
create: true`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
