Thanks. Got it working.
On 28 Mar, 2014, at 2:02 pm, Aaron Davidson wrote:
> Assuming you're using a new enough version of Spark, you should use
> spark.executor.memory to set the memory for your executors, without changing
> the driver memory. See the docs for your version of Spark.
>
>
> O
Assuming you're using a new enough version of Spark, you should use
spark.executor.memory to set the memory for your executors, without
changing the driver memory. See the docs for your version of Spark.
On Thu, Mar 27, 2014 at 10:48 PM, Tsai Li Ming wrote:
> Hi,
>
> My worker nodes have more me
Hi,
My worker nodes have more memory than the host that I’m submitting my driver
program, but it seems that SPARK_MEM is also setting the Xmx of the spark shell?
$ SPARK_MEM=100g MASTER=spark://XXX:7077 bin/spark-shell
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x7f
