cala> val newDF = spark.sql("select teamid, userid, role from teamuser")
>>> newDF: org.apache.spark.sql.DataFrame = [teamid: string, userid: string ...
>>> 1 more field]
>>>
>>> scala> val userDS: Dataset[Teamuser] = newDF.as[Teamuser]
>>> userDS:
;> +--+--+-+
>> |teamid|userid| role|
>> +------+--+-+
>> |t1|u1|role1|
>> +--+--+-+
>>
>>
>> scala> userDS.printSchema
>> root
>> |-- teamid: string (nullable = true)
>> |-- userid: s
--+--+-+
> |teamid|userid| role|
> +--+--+-+
> |t1|u1|role1|
> +--+--+-+
>
>
> scala> userDS.printSchema
> root
> |-- teamid: string (nullable = true)
> |-- userid: string (nullable = true)
> |-- role: string (nullable = tr
now I get a run time error...
error: Unable to find encoder for type stored in a Dataset. Primitive
types (Int, String, etc) and Product types (case classes) are supported by
importing spark.implicits._ Support for serializing other types will be
added in future releases.
[ERROR] val userDS:
true)
Am I missing anything?
Yong
From: shyla deshpande
Sent: Thursday, March 23, 2017 3:49 PM
To: user
Subject: Re: Converting dataframe to dataset question
I realized, my case class was inside the object. It should be defined outside
the scope of the object.
I realized, my case class was inside the object. It should be defined
outside the scope of the object. Thanks
On Wed, Mar 22, 2017 at 6:07 PM, shyla deshpande
wrote:
> Why userDS is Dataset[Any], instead of Dataset[Teamuser]? Appreciate your
> help. Thanks
>
> val spark = SparkSession
>
