Hi,
I have a case class with many columns that are Option[Int] or
Option[Array[Byte]] and such.
I would like to save it to parquet file and later read it back to my case
class too.
I found that Option[Int] when null returns 0 when the field is Null.
My question:
Is there a way to get Option[Int] from a row instead of Int from a
dataframe?
...Manas
Some more description
/*My case class*/
case class Student(name: String, age: Option[Int])
val s = new Student("Manas",Some(35))
val s1 = new Student("Manas1",None)
val student =sc.makeRDD(List(s,s1)).toDF
/*Now writing the dataframe*/
student.write.parquet("/tmp/t1")
/*Lets read it back*/
val st1 = sqlContext.read.parquet("/tmp/t1")
st1.show
+------+----+
| name| age|
+------+----+
| Manas| 35|
|Manas1|null|
+------+----+
But now I want to cast my dataframe to the dataframe[Student]. What is the
easiest way to do it?
..Manas
