This is not related to executor memory, but the extra overhead subtracted from the executor's size in order to avoid using more than the physical memory that YARN allows. That is, if you declare a 32G executor YARN lets you use 32G physical memory but your JVM heap must be significantly less than 32G max. This is the overhead factor that is subtracted for you, and it seems to need to be bigger in your case.
On Wed, Apr 15, 2015 at 10:16 AM, Brahma Reddy Battula <brahmareddy.batt...@huawei.com> wrote: > Thanks lot for your reply.. > > There is no issue with spark1.1..Following issue came when I upgrade to > spark2.0...Hence I did not decrease spark.executor.memory... > I mean to say, used same config for spark1.1 and spark1.2.. > > Is there any issue with spark1.2..? > Or Yarn will lead this..? > And why executor will not release memory, if there are tasks running..? > > > Thanks & Regards > > Brahma Reddy Battula > > > ________________________________ > From: Akhil Das [ak...@sigmoidanalytics.com] > Sent: Wednesday, April 15, 2015 2:35 PM > To: Brahma Reddy Battula > Cc: user@spark.apache.org > Subject: Re: Running beyond physical memory limits > > Did you try reducing your spark.executor.memory? > > Thanks > Best Regards > > On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula > <brahmareddy.batt...@huawei.com> wrote: >> >> Hello Sparkers >> >> >> I am newbie to spark and need help.. We are using spark 1.2, we are >> getting the following error and executor is getting killed..I seen >> SPARK-1930 and it should be in 1.2.. >> >> Any pointer to following error, like what might lead this error.. >> >> >> 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container >> [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is >> running beyond physical memory limits. Current usage: 26.0 GB of 26 GB >> physical memory used; 26.7 GB of 260 GB virtual memory used. Killing >> container. >> Dump of the process-tree for container_1429065217137_0012_01_-411041790 : >> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) >> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE >> |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 >> 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server >> -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m >> -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties >> -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native >> -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp >> -Dspark.driver.port=23204 -Dspark.random.port.max=23999 >> -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 >> -Dspark.ui.port=23000 -Dspark.random.port.min=23000 >> -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 >> org.apache.spark.executor.CoarseGrainedExecutorBackend >> akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 >> hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 >> 126843 (bash) 0 0 11603968 331 /bin/bash -c >> /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server >> -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m >> -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties >> -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native >> -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp >> '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' >> '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' >> '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' >> '-Dspark.random.port.min=23000' >> -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 >> org.apache.spark.executor.CoarseGrainedExecutorBackend >> akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 >> hadoopc1h11 10 application_1429065217137_0012 1> >> /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout >> 2> >> /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr >> | >> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) >> >> >> >> And some doubts >> >> >> 1) why executor will not release memory, if there are tasks running..? >> >> >> >> 2) is there issue from hadoop which will lead this error..? >> >> >> >> Any help , will be appreciated... >> >> >> >> >> Thanks & Regards >> >> Brahma Reddy Battula >> >> >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
