Hello Experts,
I would need your advice in resolving the below issue when I am trying to
retrieving the data from a dataframe.
Can you please let me know where I am going wrong.
code :
// create the dataframe by parsing the json
// Message Helper describes the JSON Struct
//data out is the json string received from Streaming Engine.
val dataDF = sparkSession.createDataFrame(dataOut, MessageHelper.sqlMapping)
dataDF.printSchema()
/* -- out put of dataDF.printSchema
root
|-- messageID: string (nullable = true)
|-- messageType: string (nullable = true)
|-- meta: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- messageParsedTimestamp: string (nullable = true)
| | |-- ipaddress: string (nullable = true)
|-- messageData: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- packetID: string (nullable = true)
| | |-- messageID: string (nullable = true)
| | |-- unixTime: string (nullable = true)
*/
dataDF.createOrReplaceTempView("message")
val routeEventDF=sparkSession.sql("select messageId
,messageData.unixTime,messageData.packetID, messageData.messageID from
message")
routeEventDF.show
Error on routeEventDF.show
Caused by: java.lang.RuntimeException:
org.apache.spark.sql.catalyst.expressions.GenericRow is not a valid
external type for schema of
array<struct<messageParsedTimestamp:string,ipaddress:string,port:string,message:string>>>>
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.evalIfFalseExpr14$(Unknown
Source)
Appreciate your help
Best Regards
Arnav Kumar.
