Thanks lot for your reply.. There is no issue with spark1.1..Following issue came when I upgrade to spark2.0...Hence I did not decrease spark.executor.memory... I mean to say, used same config for spark1.1 and spark1.2..
Is there any issue with spark1.2..? Or Yarn will lead this..? And why executor will not release memory, if there are tasks running..? Thanks & Regards Brahma Reddy Battula ________________________________ From: Akhil Das [ak...@sigmoidanalytics.com] Sent: Wednesday, April 15, 2015 2:35 PM To: Brahma Reddy Battula Cc: user@spark.apache.org Subject: Re: Running beyond physical memory limits Did you try reducing your spark.executor.memory? Thanks Best Regards On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula <brahmareddy.batt...@huawei.com<mailto:brahmareddy.batt...@huawei.com>> wrote: Hello Sparkers I am newbie to spark and need help.. We are using spark 1.2, we are getting the following error and executor is getting killed..I seen SPARK-1930 and it should be in 1.2.. Any pointer to following error, like what might lead this error.. 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory used; 26.7 GB of 260 GB virtual memory used. Killing container. Dump of the process-tree for container_1429065217137_0012_01_-411041790 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp -Dspark.driver.port=23204 -Dspark.random.port.max=23999 -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 -Dspark.ui.port=23000 -Dspark.random.port.min=23000 -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler<http://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler> 3 hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 126843 (bash) 0 0 11603968 331 /bin/bash -c /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' '-Dspark.random.port.min=23000' -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler<http://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler> 3 hadoopc1h11 10 application_1429065217137_0012 1> /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout 2> /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr | org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) And some doubts 1) why executor will not release memory, if there are tasks running..? 2) is there issue from hadoop which will lead this error..? Any help , will be appreciated... Thanks & Regards Brahma Reddy Battula
