Hi, Fletcher.
case class can help construct complex structure.
and also, RDD, StructType and StructureField are helpful if you need.
However,
the code is a little confusing,
source.map{ row => {
val key = row(0)
val buff = new ArrayBuffer[Row]()
buff += row
(key,buff)
sorry here's the whole code
val source =
spark.read.format("parquet").load("/emrdata/sources/very_large_ds")
implicit val mapEncoder =
org.apache.spark.sql.Encoders.kryo[(Any,ArrayBuffer[Row])]
source.map{ row => {
val key = row(0)
val buff = new ArrayBuffer[Row]()
buff += row
