agreed. need to be enhanced!
HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Mar 2023 at 19:15, Ismail Yenigul <ismailyeni...@gmail.com> wrote: > Hi Mich, > > The issue is here there is no parameter to set executor pod request memory > value. > Currently we have only one parameter which is spark.executor.memory and it > set pod resources limit and requests. > > Mich Talebzadeh <mich.talebza...@gmail.com>, 10 Mar 2023 Cum, 22:04 > tarihinde şunu yazdı: > >> Yes, both EKS and GKE (Google) are on 3.1.2 so I am not sure those >> parameters will work :( >> >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Fri, 10 Mar 2023 at 19:01, Ismail Yenigul <ismailyeni...@gmail.com> >> wrote: >> >>> Hi Mich, >>> >>> it is on AWS EKS >>> >>> >>> Mich Talebzadeh <mich.talebza...@gmail.com>, 10 Mar 2023 Cum, 21:11 >>> tarihinde şunu yazdı: >>> >>>> I forgot top ask which k8s cluster are you using, assuming some clod >>>> vendor >>>> >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <ismailyeni...@gmail.com> >>>> wrote: >>>> >>>>> and If you look at the code >>>>> >>>>> >>>>> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194 >>>>> >>>>> .editOrNewResources() >>>>> .addToRequests("memory", executorMemoryQuantity) >>>>> .addToLimits("memory", executorMemoryQuantity) >>>>> .addToRequests("cpu", executorCpuQuantity) >>>>> .addToLimits(executorResourceQuantities.asJava) >>>>> .endResources() >>>>> >>>>> addToRequests and addToLimits for memory have the same value. >>>>> maybe it is by design. but can I set custom values for them if I use >>>>> podtemplate? >>>>> >>>>> >>>>> >>>>> Ismail Yenigul <ismailyeni...@gmail.com>, 10 Mar 2023 Cum, 20:52 >>>>> tarihinde şunu yazdı: >>>>> >>>>>> Hi, >>>>>> using spark version v.3.1.2 >>>>>> >>>>>> spark.executor.memory is set. >>>>>> But the problem is not setting spark.executor.memory, the problem is >>>>>> that whatever value I set spark.executor.memory, >>>>>> spark executor pod has the same value for resources.limit.memory and >>>>>> resources.request.memory. >>>>>> I want to be able to set different values for them. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Mich Talebzadeh <mich.talebza...@gmail.com>, 10 Mar 2023 Cum, 20:44 >>>>>> tarihinde şunu yazdı: >>>>>> >>>>>>> What are those currently set in spark-submit and which spark version >>>>>>> on k8s >>>>>>> >>>>>>> --conf spark.driver.memory=2000m \ >>>>>>> --conf spark.executor.memory=2000m \ >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>> for any loss, damage or destruction of data or any other property which >>>>>>> may >>>>>>> arise from relying on this email's technical content is explicitly >>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>> damages >>>>>>> arising from such loss, damage or destruction. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul < >>>>>>> ismailyeni...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> There is a cpu parameter to set spark executor on k8s >>>>>>>> spark.kubernetes.executor.limit.cores and >>>>>>>> spark.kubernetes.executor.request.cores >>>>>>>> but there is no parameter to set memory request different then >>>>>>>> limits memory (such as spark.kubernetes.executor.request.memory) >>>>>>>> For that reason, >>>>>>>> spark.executor.memory is assigned to requests.memory and >>>>>>>> limits.memory like the following >>>>>>>> >>>>>>>> Limits: >>>>>>>> memory: 5734MiRequests: >>>>>>>> cpu: 4 >>>>>>>> memory: 5734Mi >>>>>>>> >>>>>>>> >>>>>>>> Is there any special reason to not have >>>>>>>> spark.kubernetes.executor.request.memory parameter? >>>>>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter >>>>>>>> to set smaller memory request than the memory limit in pod template >>>>>>>> file? >>>>>>>> >>>>>>>> >>>>>>>> Limits: >>>>>>>> memory: 5734MiRequests: >>>>>>>> cpu: 4 >>>>>>>> memory: 1024Mi >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>>
