Hi all, I was thinking about whether it would make sense to support ConfigMap as a volume type in Spark’s generic Kubernetes volume configuration.
We operate a multi-tenant Spark-on-Kubernetes platform and use ConfigMaps to supply runtime configuration for in-house extensions and platform integrations. We would also like application owners to add their own ConfigMaps or override selected platform defaults without needing to understand or maintain the platform’s pod configuration. Currently, we handle these mounts through driver and executor pod templates. Since there is one template per pod type, adding user-specific mounts means merging templates ourselves or asking users to preserve platform-specific details in their templates(which we really want to avoid). It seems like ConfigMaps could fit into the existing named-volume API: spark.kubernetes.driver.volumes.configMap.<volume>.options.configMapName=<configmap> spark.kubernetes.driver.volumes.configMap.<volume>.mount.path=<path> spark.kubernetes.executor.volumes.configMap.<volume>.options.configMapName=<configmap> spark.kubernetes.executor.volumes.configMap.<volume>.mount.path=<path> This would let us supply platform defaults through Spark properties while allowing users to add separate named mounts or select a different ConfigMap for an existing mount, where permitted. Our platform would handle property precedence before submission, without having to merge complete pod templates. The scope would be mounting pre-existing ConfigMaps, with no change to how Spark loads submission-time properties. While looking into this, I came across [SPARK-25500](https://issues.apache.org/jira/browse/SPARK-25500) and the associated [PR #22536](https://github.com/apache/spark/pull/22536), which proposed similar functionality in 2018. It was closed based on the position that additional pod customization should use pod templates. Would maintainers still prefer pod templates for this use case, or would it make sense to revisit ConfigMap support as another type in the existing volume API? If there is interest, I’d be happy to work on the implementation. I’d also appreciate guidance on the desired initial scope. Thanks, Tamir --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
