>From my experience with spark 1.3.1 you can also set spark.executor.memoryOverhead to about 7-10% of your spark.executor.memory. Total of which will be requested for a Yarn container.
On Tue, Jan 26, 2016 at 4:20 AM, Xiaoyu Ma <[email protected]> wrote: > Hi all, > I saw spark 1.6 has new off heap settings: spark.memory.offHeap.size > The doc said we need to shrink on heap size accordingly. But on Yarn > on-heap and yarn limit is set all together via spark.executor.memory (jvm > opts for memory is not allowed according to doc), how can we set executor > JVM heap size and Yarn container memory limit differently? > > Thanks. > > 马晓宇 > [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] <https://twitter.com/Xactly> [image: Facebook] <https://www.facebook.com/XactlyCorp> [image: YouTube] <http://www.youtube.com/xactlycorporation>
