For anyone interested, the problem ended up being that in some rare cases,
the value from the pair RDD on the right side of the left outer join was
Java's null. The Spark optionToOptional method attempted to apply Some()
to null, which caused the NPE to be thrown.
The lesson is to filter out any
Hi Ted,
I am working on replicating the problem on a smaller scale.
I saw that Spark 2.0 is moving to Java 8 Optional instead of Guava
Optional, but in the meantime I'm stuck with 1.6.1.
-Adam
On Fri, May 6, 2016 at 9:40 AM, Ted Yu wrote:
> Is it possible to write a short test which exhibits
Is it possible to write a short test which exhibits this problem ?
For Spark 2.0, this part of code has changed:
[SPARK-4819] Remove Guava's "Optional" from public API
FYI
On Fri, May 6, 2016 at 6:57 AM, Adam Westerman wrote:
> Hi,
>
> I’m attempting to do a left outer join in Spark, and I’m
Hi,
I’m attempting to do a left outer join in Spark, and I’m getting an NPE
that appears to be due to some Spark Java API bug. (I’m running Spark 1.6.0
in local mode on a Mac).
For a little background, the left outer join returns all keys from the left
side of the join regardless of whether or no
