Yeah, as Sameer commented, there is unfortunately not an equivalent `SPARK_MASTER_MEMORY` that you can set. You can work around this by starting the master and the slaves separately with different settings of SPARK_DAEMON_MEMORY each time.
AFAIK there haven't been any major changes in the standalone master in 1.1.0, so I don't see an immediate explanation for what you're observing. In general the Spark master doesn't use that much memory, and even if there are many applications it will discard the old ones appropriately, so unless you have a ton (like thousands) of concurrently running applications connecting to it there's little likelihood for it to OOM. At least that's my understanding. -Andrew 2014-10-22 15:51 GMT-07:00 Sameer Farooqui <same...@databricks.com>: > Hi Keith, > > Would be helpful if you could post the error message. > > Are you running Spark in Standalone mode or with YARN? > > In general, the Spark Master is only used for scheduling and it should be > fine with the default setting of 512 MB RAM. > > Is it actually the Spark Driver's memory that you intended to change? > > > > *++ If in Standalone mode ++* > You're right that SPARK_DAEMON_MEMORY set the memory to allocate to the > Spark Master, Worker and even HistoryServer daemons together. > > SPARK_WORKER_MEMORY is slightly confusing. In Standalone mode, it is the > amount of memory that a worker advertises as available for drivers to > launch executors. The sum of the memory used by executors spawned from a > worker cannot exceed SPARK_WORKER_MEMORY. > > Unfortunately, I'm not aware of a way to set the memory for Master and > Worker individually, other than launching them manually. You can also try > setting the config differently on each machine's spark-env.sh file. > > > *++ If in YARN mode ++* > In YARN, there is no setting for SPARK_DAEMON_MEMORY. Therefore this is > only in the Standalone documentation. > > Remember that in YARN mode there is no Spark Worker, instead the YARN > NodeManagers launches the Executors. And in YARN, there is no need to run a > Spark Master JVM (since the YARN ResourceManager takes care of the > scheduling). > > So, with YARN use SPARK_EXECUTOR_MEMORY to set the Executor's memory. And > use SPARK_DRIVER_MEMORY to set the Driver's memory. > > Just an FYI - for compatibility's sake, even in YARN mode there is a > setting for SPARK_WORKER_MEMORY, but this has been deprecated. If you do > set it, it just does the same thing as setting SPARK_EXECUTOR_MEMORY would > have done. > > > - Sameer > > > On Wed, Oct 22, 2014 at 1:46 PM, Keith Simmons <ke...@pulse.io> wrote: > >> We've been getting some OOMs from the spark master since upgrading to >> Spark 1.1.0. I've found SPARK_DAEMON_MEMORY, but that also seems to >> increase the worker heap, which as far as I know is fine. Is there any >> setting which *only* increases the master heap size? >> >> Keith >> > >
