Hi Nick,
The scope of the PR I submitted is reduced because we can't make sure if it
is really the root cause of the error you faced. You can check out the
discussion on the PR. So I can just change the assert in the code as shown
in the PR.
If you can have a repro, we can go back to see if it i
Subject:Re: Reduce memory usage of UnsafeInMemorySorter
Unfortunately, I don't have a repro, and I'm only seeing this at scale.
But I was able to get around the issue by fiddling with the distribution
of my data before asking GraphFrames to process it. (I think that's where
.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L156
>
> Regards,
> Kazuaki Ishizaki
>
>
>
> From:Reynold Xin
> To:Nicholas Chammas
> Cc: Spark dev list
> D
org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L156
Regards,
Kazuaki Ishizaki
From: Reynold Xin
To: Nicholas Chammas
Cc: Spark dev list
Date: 2016/12/07 14:27
Subject: Re: Reduce memory usage of UnsafeInMemorySorter
This is not supposed to happen. Do
This is not supposed to happen. Do you have a repro?
On Tue, Dec 6, 2016 at 6:11 PM, Nicholas Chammas wrote:
> [Re-titling thread.]
>
> OK, I see that the exception from my original email is being triggered
> from this part of UnsafeInMemorySorter:
>
> https://github.com/apache/spark/blob/v2.0.
