Hi, There is a cpu parameter to set spark executor on k8s spark.kubernetes.executor.limit.cores and spark.kubernetes.executor.request.cores but there is no parameter to set memory request different then limits memory (such as spark.kubernetes.executor.request.memory) For that reason, spark.executor.memory is assigned to requests.memory and limits.memory like the following
Limits: memory: 5734MiRequests: cpu: 4 memory: 5734Mi Is there any special reason to not have spark.kubernetes.executor.request.memory parameter? and can I use spark.kubernetes.executor.podTemplateFile parameter to set smaller memory request than the memory limit in pod template file? Limits: memory: 5734MiRequests: cpu: 4 memory: 1024Mi Thanks