Got it - thanks! On Sat, Dec 6, 2014 at 14:56 Arun Ahuja <aahuj...@gmail.com> wrote:
> Hi Denny, > > This is due the spark.yarn.memoryOverhead parameter, depending on what > version of Spark you are on the default of this may differ, but it should > be the larger of 1024mb per executor or .07 * executorMemory. > > When you set executor memory, the yarn resource request is executorMemory > + yarnOverhead. > > - Arun > > On Sat, Dec 6, 2014 at 4:27 PM, Denny Lee <denny.g....@gmail.com> wrote: > >> This is perhaps more of a YARN question than a Spark question but i was >> just curious to how is memory allocated in YARN via the various >> configurations. For example, if I spin up my cluster with 4GB with a >> different number of executors as noted below >> >> 4GB executor-memory x 10 executors = 46GB (4GB x 10 = 40 + 6) >> 4GB executor-memory x 4 executors = 19GB (4GB x 4 = 16 + 3) >> 4GB executor-memory x 2 executors = 10GB (4GB x 2 = 8 + 2) >> >> The pattern when observing the RM is that there is a container for each >> executor and one additional container. From the basis of memory, it looks >> like there is an additional (1GB + (0.5GB x # executors)) that is allocated >> in YARN. >> >> Just wondering why is this - or is this just an artifact of YARN itself? >> >> Thanks! >> >> >
