Hi,
I'm trying to call flatMap on a Dataframe with Spark 2.0 (rc5).
The code is the following:
var data = spark.read.parquet(fileName).flatMap(x => List(x))
Of course it's an overly simplified example, but the result is the same.
The dataframe schema goes from this:
root
|-- field1: d
k-1-6
How did you setup your encoder?
- Mail original -
De: "Sun Rui"
À: "Julien Nauroy"
Cc: user@spark.apache.org
Envoyé: Samedi 23 Juillet 2016 15:55:21
Objet: Re: Using flatMap on Dataframes with Spark 2.0
I did a try. the schema after flatMap is the same, wh
= df1.flatMap(x => List(x))
df1.printSchema() // newCol has disappeared
Any idea what I could be doing wrong? Why would newCol disappear?
Cheers,
Julien
- Mail original -
De: "Julien Nauroy"
À: "Sun Rui"
Cc: user@spark.apache.org
Envoyé: Samedi 23 Ju
", df1.col("anyExistingCol"))
df1.printSchema() // here newCol exists
implicit val encoder: ExpressionEncoder[Row] = RowEncoder(df1.schema)
df1 = df1.flatMap(x => List(x))
df1.printSchema() // newCol still exists!
Julien
- Mail original -
De: "Julien
