Hi I would like to know more about typeface aggregations in spark. http://stackoverflow.com/questions/40596638/inquiries-about-spark-2-0-dataset/40602882?noredirect=1#comment70139481_40602882 An example of these is https://blog.codecentric.de/en/2016/07/spark-2-0-datasets-case-classes/ ds.groupByKey(body => body.color)
does "myDataSet.map(foo.someVal) is type safe but as any Dataset operation uses RDD and compared to DataFrame operations there is a significant overhead. Let's take a look at a simple example:" hold true e.g. will type safe aggregation require the deserialisation of the full objects as displayed for ds.map(_.foo).explain ? Kind regards, Georg -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Clarification-about-typesafe-aggregations-tp20459.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
