I am creating a spark structured streaming application and I have a streaming
dataframe which has the below data in it.
{
"name":"sensor1",
"time":"2020-11-27T01:01:00",
"sensorvalue":11.0,
"tag1":"tagvalue"
}
I would like to convert that dataframe into below format.
{
"name":"sensor1",
"value-array":
[
{
"time":"2020-11-27T01:01:00",
"sensorvalue":11.0,
"tag1":"tagvalue"
}
]
}
I tried using mapPartition() / map() method, where I get a row Object and I
tried creating another dataframe in the expected format.
I am able to get the values from row object using row(0) or row(1). But, Is
it possible to put a POJO/schema for that row object (Using Row Encoders())
? That way instead of using row(0), we can use row.getName() ?
val mapDF = incomingDF.map(row =>{
val name = row(0).toString
/* val name = row.getName */
})
I tried using collect_list() function. Since its an aggregate function, I
couldn't use "append" as output mode in my streaming application.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
