Metadata is not propagating with Dataset.map()

2017-01-16 Thread Matthew Tovbin
Hello, It seems that metadata is not propagating when using Dataset.map(). Is there a workaround? Below are the steps to reproduce: import spark.implicits._ val columnName = "col1" val meta = new MetadataBuilder().putString("foo", "bar").build() val schema = StructType(Array(StructField(columnNa

Metadata is not propagating with Dataset.map()

2017-01-16 Thread tovbinm
Hello, It seems that metadata is not propagating when using Dataset.map(). Is there a workaround? Below are the steps to reproduce: import spark.implicits._ val columnName = "col1" val meta = new MetadataBuilder().putString("foo", "bar").build() val schema = StructType(Array(StructField(columnNa