Re: Questions regarding memory usage

2014-09-12 Thread Sean Owen
On Thu, Sep 11, 2014 at 10:17 PM, Tom wrote: > If I set SPARK_DRIVER_MEMORY to x GB, Spark reports > /14/09/11 15:36:41 INFO MemoryStore: MemoryStore started with capacity > ~0.55*x GB/ > *Question:* > Does this relate to spark.storage.memoryFraction (default 0.6), and is the > other 0.4 used by s

Questions regarding memory usage

2014-09-11 Thread Tom
Hi All, I currently have 3 questions regarding memory usage: 1) Regarding overall memory usage: If I set SPARK_DRIVER_MEMORY to x GB, Spark reports /14/09/11 15:36:41 INFO MemoryStore: MemoryStore started with capacity ~0.55*x GB/ *Question:* Does this relate to spark.storage.memoryFraction