RE: Spark consumes more memory

2017-05-11 Thread Anantharaman, Srinatha (Contractor)
, May 11, 2017 1:34 PM To: Anantharaman, Srinatha (Contractor) ; user Subject: Re: Spark consumes more memory I would try to track down the "no space left on device" - find out where that originates from, since you should be able to allocate 10 executors with 4 cores and 15GB RAM

Re: Spark consumes more memory

2017-05-11 Thread Rick Moritz
I would try to track down the "no space left on device" - find out where that originates from, since you should be able to allocate 10 executors with 4 cores and 15GB RAM each quite easily. In that case,you may want to increase overhead, so yarn doesn't kill your executors. Check that no local driv