Nauroy"
À: "Sun Rui"
Cc: user@spark.apache.org
Envoyé: Dimanche 24 Juillet 2016 12:43:42
Objet: Re: Using flatMap on Dataframes with Spark 2.0
Hi again,
Just another strange behavior I stumbled upon. Can anybody reproduce it?
Here's the code snippet in scala:
var df1 =
iginal -
De: "Sun Rui"
À: "Julien Nauroy"
Cc: user@spark.apache.org
Envoyé: Samedi 23 Juillet 2016 16:27:43
Objet: Re: Using flatMap on Dataframes with Spark 2.0
You should use :
import org.apache.spark.sql.catalyst.encoders.RowEncoder
val df = spark.read.parq
-a-dataset-in-spark-1-6
>
> <http://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-a-dataset-in-spark-1-6>
>
> How did you setup your encoder?
>
>
> De: "Sun Rui"
> À: "Julien Nauroy"
> Cc: user@spark.apache.org
> Envoyé: Sam
k-1-6
How did you setup your encoder?
- Mail original -
De: "Sun Rui"
À: "Julien Nauroy"
Cc: user@spark.apache.org
Envoyé: Samedi 23 Juillet 2016 15:55:21
Objet: Re: Using flatMap on Dataframes with Spark 2.0
I did a try. the schema after flatMap is the same, wh
I did a try. the schema after flatMap is the same, which is expected.
What’s your Row encoder?
> On Jul 23, 2016, at 20:36, Julien Nauroy wrote:
>
> Hi,
>
> I'm trying to call flatMap on a Dataframe with Spark 2.0 (rc5).
> The code is the following:
> var data = spark.read.parquet(fileName).fla
Hi,
I'm trying to call flatMap on a Dataframe with Spark 2.0 (rc5).
The code is the following:
var data = spark.read.parquet(fileName).flatMap(x => List(x))
Of course it's an overly simplified example, but the result is the same.
The dataframe schema goes from this:
root
|-- field1: d
