ruiliang created SPARK-51480:
--------------------------------
Summary: saprk jvm xmx and container momory need to be set
separately
Key: SPARK-51480
URL: https://issues.apache.org/jira/browse/SPARK-51480
Project: Spark
Issue Type: Improvement
Components: Spark Submit
Affects Versions: 3.5.5
Reporter: ruiliang
h2. [Elastic Memory Feature|
[https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManagerCGroupsMemory.html#:~:text=Elastic%20Memory%20Feature]
h2. ]
[spark set
xmx|https://spark.apache.org/docs/latest/configuration.html#:~:text=3.0.0-,spark.executor.extraJavaOptions,-(none)]
Currently, mapreduce can be specified separately
{code:java}
mapreduce.{reduce|map}.memory-mb={xx}
mapreduce.{reduce|map}.java.opts=-Xmx{xx}m{code}
[spark That should be removed here as
well|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L570-L572]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
