In Spark 1.6 there is an experimental new features called Datasets. You
can call df.as[Student] and it should do what you want. Would love any
feedback you have if you get a chance to try it out (we'll hopefully
publish a preview release next week).
On Mon, Nov 2, 2015 at 9:30 PM, manas kar <poorinsp...@gmail.com> wrote:
> Hi,
> I have a case class with many columns that are Option[Int] or
> Option[Array[Byte]] and such.
> I would like to save it to parquet file and later read it back to my case
> class too.
> I found that Option[Int] when null returns 0 when the field is Null.
> My question:
> Is there a way to get Option[Int] from a row instead of Int from a
> dataframe?
>
> ...Manas
>
> Some more description
>
> /*My case class*/
> case class Student(name: String, age: Option[Int])
>
> val s = new Student("Manas",Some(35))
> val s1 = new Student("Manas1",None)
> val student =sc.makeRDD(List(s,s1)).toDF
>
> /*Now writing the dataframe*/
> student.write.parquet("/tmp/t1")
>
> /*Lets read it back*/
> val st1 = sqlContext.read.parquet("/tmp/t1")
> st1.show
>
> +------+----+
> | name| age|
> +------+----+
> | Manas| 35|
> |Manas1|null|
> +------+----+
>
> But now I want to cast my dataframe to the dataframe[Student]. What is the
> easiest way to do it?
>
> ..Manas
>
