[jira] [Created] (SPARK-51435) spark submit executor jvm Xmx and yarn container container_memory Whether it can be set separately

[ruiliang (Jira)]

ruiliang created SPARK-51435:
--------------------------------

             Summary: spark submit executor jvm Xmx and yarn container 
container_memory  Whether it can be set separately
                 Key: SPARK-51435
                 URL: https://issues.apache.org/jira/browse/SPARK-51435
             Project: Spark
          Issue Type: Question
          Components: Spark Submit
    Affects Versions: 3.2.1
            Reporter: ruiliang


spark submit executor jvm Xmx and yarn container container_memory  Whether it 
can be set separately

yarn submit doc

[YarnApplications submit  
args|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#:~:text=//%20Set%20Xmx%20based%20on%20am%20memory%20size%0Avargs.add(%22%2DXmx%22%20%2B%20amMemory%20%2B%20%22m%22)%3B%0A//%20Set%20class%20name%0Avargs.add(appMasterMainClass)%3B%0A//%20Set%20params%20for%20Application%20Master%0Avargs.add(%22%2D%2Dcontainer_memory%20%22%20%2B%20String.valueOf(containerMemory))%3B]
 

 
h2. [YARN Elastic Memory Feature 
|https://hadoop.apache.org/docs/r3.3.0/hadoop-yarn/hadoop-yarn-site/NodeManagerCGroupsMemory.html#:~:text=Elastic%20Memory%20Feature,the%20OOM%20condition]

container Elastic Memory is required but spark submit executor defaults to 
set-Xmx resulting in jvm oom

 

My needs

https://issues.apache.org/jira/projects/YARN/issues/YARN-11789?filter=reportedbyme

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org