Re: DF.intersection issue in 1.5

Thanks for pointing this out. https://issues.apache.org/jira/browse/SPARK-10539 We will fix this for Spark 1.5.1. On Thu, Sep 10, 2015 at 6:16 AM, Nitay Joffe wrote: > The following fails for me in Spark 1.5: > https://gist.github.com/nitay/d08cb294ccf00b80c49a > Specifically, it returns 1 inst

DF.intersection issue in 1.5

The following fails for me in Spark 1.5: https://gist.github.com/nitay/d08cb294ccf00b80c49a Specifically, it returns 1 instead of 100 (in both versions). When I print out the contents (i.e. collect()) I see all 100 items, yet the count returns 1. This works in 1.3 and 1.4. Any ideas what's going