Re: can not join dataset with itself

2016-04-08 Thread JH P
I’m using Spark 1.6.1 Class is case class DistinctValues(statType: Int, dataType: Int, _id: Int, values: Array[(String, Long)], numOfMembers: Int,category: String) and error for newGnsDS.joinWith(newGnsDS, $"dataType”) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot r

Re: can not join dataset with itself

2016-04-08 Thread Ted Yu
Looks like you're using Spark 1.6.x What error(s) did you get for the first two joins ? Thanks On Fri, Apr 8, 2016 at 3:53 AM, JH P wrote: > Hi. I want a dataset join with itself. So i tried below codes. > > 1. newGnsDS.joinWith(newGnsDS, $"dataType”) > > 2. newGnsDS.as("a").joinWith(newGnsDS.