If you're pointing at the 336MB, then it's not really related any of the items you cite here. This is the memory managed internally by MemoryStore. The blog post refers to the legacy memory manager. You can see a bit of how it works in the code, but this is the sum of the on-heap and off-heap memory it can manage. See the memory config docs, however, to understand what user-facing settings you can make; you don't really need to workk about this value.
mapreduce settings are irrelevant to Spark. Spark doesn't pay attention to the YARN settings, but YARN does. It enforces them, yes. It is not exempt from YARN. 896MB is correct there. yarn-client mode does not ignore driver properties, no. On Sat, Nov 12, 2016 at 2:18 AM Elkhan Dadashov <elkhan8...@gmail.com> wrote: > Hi, > > Spark website <http://spark.apache.org/docs/latest/running-on-yarn.html> > indicates > default spark properties as like this: > I did not override any properties in spark-defaults.conf file, but when I > launch Spark in YarnClient mode: > > spark.driver.memory 1g > spark.yarn.am.memory 512m > spark.yarn.am.memoryOverhead : max(spark.yarn.am.memory * 0.10, 384m) > spark.yarn.driver.memoryOverhead : max(spark.driver.memory * 0.10, 384m) > > I launch Spark job via SparkLauncher#startApplication() in *Yarn-client > mode from the Map task of Hadoop job*. > > *My cluster settings*: > yarn.scheduler.minimum-allocation-mb 256 > yarn.scheduler.maximum-allocation-mb 2048 > yarn.app.mapreduce.am.resource.mb 512 > mapreduce.map.memory.mb 640 > mapreduce.map.java.opts -Xmx400m > yarn.app.mapreduce.am.command-opts -Xmx448m > > *Logs of Spark job*: > > INFO Client: Verifying our application has not requested more than the > maximum memory capability of the cluster (2048 MB per container) > INFO Client: Will allocate *AM container*, with 896 MB memory including > 384 MB overhead > > INFO MemoryStore: MemoryStore started with capacity 366.3 MB > > ./application_1478727394310_0005/container_1478727394310_0005_01_000002/stderr:INFO: > 16/11/09 14:18:42 INFO BlockManagerMasterEndpoint: Registering block > manager <machine-ip>:57246 with *366.3* MB RAM, BlockManagerId(driver, > <machine-ip>, 57246) > > *Questions*: > 1) How is driver memory calculated ? > > How did Spark decide for 366 MB for driver based on properties described > above ? > > I thought the memory allocation is based on this formula ( > https://www.altiscale.com/blog/spark-on-hadoop/ ): > > "Runtime.getRuntime.maxMemory * memoryFraction * safetyFraction ,where > memoryFraction=0.6, and safetyFraction=0.9. This is 1024MB x 0.6 x 0.9 = > 552.96MB. However, 552.96MB is a little larger than the value as shown in > the log. This is because of the runtime overhead imposed by Scala, which is > usually around 3-7%, more or less. If you do the calculation using 982MB x > 0.6 x 0.9, (982MB being approximately 4% of 1024) then you will derive the > number 530.28MB, which is what is indicated in the log file after rounding > up to 530.30MB." > > 2) If Spark job is launched from the Map task via > SparkLauncher#startApplication() will driver memory respect > (mapreduce.map.memory.mb and mapreduce.map.java.opts) OR > (yarn.scheduler.maximum-allocation-mb) when launching Spark Job as child > process ? > > The confusion is, as SparkSubmit is a new JVM process - because it is > launched as child process of the map task, and it does not depend on Yarn > configs. But not obeying any limits (if this is the case), will make things > tricky on NodeManager reporting back memory usage. > > 3) Is this correct formula for calculating AM memory ? > > For AM it matches to this formula calculation ( > https://www.altiscale.com/blog/spark-on-hadoop/ ):how much memory to > allocate to the AM: amMemory + amMemoryOverhead > amMemoryOverhead is set to 384MB via spark.yarn.driver.memoryOverhead. > args.amMemory is fixed at 512MB by Spark when it’s running in yarn-client > mode. Adding 384MB of overhead to 512MB provides the 896MB figure requested > by Spark. > > 4) For Spark Yarn-client mode, are all spark.driver properties ignored, > and only spark.yarn.am properties used ? > > Thanks. > > > > >
