Hi, using spark version v.3.1.2 spark.executor.memory is set. But the problem is not setting spark.executor.memory, the problem is that whatever value I set spark.executor.memory, spark executor pod has the same value for resources.limit.memory and resources.request.memory. I want to be able to set different values for them.
Mich Talebzadeh <mich.talebza...@gmail.com>, 10 Mar 2023 Cum, 20:44 tarihinde şunu yazdı: > What are those currently set in spark-submit and which spark version on k8s > > --conf spark.driver.memory=2000m \ > --conf spark.executor.memory=2000m \ > > HTH > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <ismailyeni...@gmail.com> > wrote: > >> Hi, >> >> There is a cpu parameter to set spark executor on k8s >> spark.kubernetes.executor.limit.cores and >> spark.kubernetes.executor.request.cores >> but there is no parameter to set memory request different then limits >> memory (such as spark.kubernetes.executor.request.memory) >> For that reason, >> spark.executor.memory is assigned to requests.memory and limits.memory >> like the following >> >> Limits: >> memory: 5734MiRequests: >> cpu: 4 >> memory: 5734Mi >> >> >> Is there any special reason to not have >> spark.kubernetes.executor.request.memory parameter? >> and can I use spark.kubernetes.executor.podTemplateFile parameter to set >> smaller memory request than the memory limit in pod template file? >> >> >> Limits: >> memory: 5734MiRequests: >> cpu: 4 >> memory: 1024Mi >> >> >> Thanks >> >>
