Hi Josh,
I see, I haven't heard folks using larger JVM heap size than you mentioned
(30gb), but in your scenario what you're proposing does make sense.
I've created SPARK-5095 and we can continue our discussion about how to
address this.
Tim
On Mon, Jan 5, 2015 at 1:22 AM, Josh Devins wrote:
Hey Tim, sorry for the delayed reply, been on vacation for a couple weeks.
The reason we want to control the number of executors is that running
executors with JVM heaps over 30GB causes significant garbage
collection problems. We have observed this through much
trial-and-error for jobs that are a
