Hi All, I’m on a platform team running Spark as a Service, which uses the Spark K8s Operator to launch Spark jobs. Under heavy load (300+ concurrent submissions), the operator becomes a bottleneck, each spark submit takes ~20 seconds, leading to 10+ minute wait times for jobs at the end of the queue.
To reduce latency, we’ve already: 1. Separated the thread pools/queues for submission (onAdd) and controller updates (onUpdate). 2. Switched to programmatic spark-submit, launching driver pods directly without forking a JVM. Now, we are looking into horizontal scaling for the operator and considering a *Leader-Orchestrated Shard Assignment* model: - A leader instance handles initial onAdd events and assigns a shard label to each SparkApplication. - Worker instances watch and process only their assigned shard. - The leader manages worker heartbeats (via ConfigMap/lease) and reassigns shards if a worker fails. New workers register dynamically to scale out. Has anyone implemented a similar sharding strategy for the Spark Operator, or are there better alternatives used in production? Thanks, Prabhu Joseph
