Had the same issue, it seems that it is simply not possible - https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L195
There's also a Jira ticket - https://issues.apache.org/jira/browse/SPARK-37358?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22kubernetes%20request%20limit%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC [https://opengraph.githubassets.com/0507edbcc6647cf1b2753e345f2e763cefb4dacc64818621fa683d29e0ceee20/apache/spark]<https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L195> spark/BasicExecutorFeatureStep.scala at master · apache/spark<https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L195> Apache Spark - A unified analytics engine for large-scale data processing - spark/BasicExecutorFeatureStep.scala at master · apache/spark github.com ________________________________ From: Yosr Kchaou <yosr.kchao...@gmail.com> Sent: Wednesday, December 7, 2022 10:19 AM To: user@spark.apache.org <user@spark.apache.org> Subject: [EXTERNAL] [SPARK Memory management] Does Spark support setting limits/requests for driver/executor memory ? ATTENTION: This email originated from outside of GM. Hello, We are running Spark on Kubernetes and noticed that driver/executors use the same value for memory request and memory limit. We see that limits/requests can be set only for cpu using the following options: spark.kubernetes.{driver/executor}.limit.cores and spark.kubernetes.{driver/executor}.request.cores. In our case, it is useful to set a memory request lower than the memory limit as it helps us better manage our cluster resources. Is there a way to define limits/requests for memory resources for driver/executor pods ? Thanks in advance. Yosr
