If you invoked the shuffling that eats a large amount of execution memory,
it possibly swept away
cached RDD blocks because the memory for the shuffling run short.
Please see:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L32
//
Each executor on the screenshot has 25GB memory remaining . What was the
reason to store 170-500 MB to disk if executor has 25GB memory available?
On Thu, May 12, 2016 at 5:12 PM, Takeshi Yamamuro
wrote:
> Hi,
>
> Not sure this is a correct answer though, seems `UnifiedMemoryManager`
> spills
>
Hi,
Not sure this is a correct answer though, seems `UnifiedMemoryManager`
spills
some blocks of RDDs into disk when execution memory runs short.
// maropu
On Fri, May 13, 2016 at 6:16 AM, Alexander Pivovarov
wrote:
> Hello Everyone
>
> I use Spark 1.6.0 on YARN (EMR-4.3.0)
>
> I use MEMORY_A
