I want to parse the Struct of data dynamically , then write data to delta lake
, I think it can automatically merge scheme.
2019-09-17
lk_spark
发件人：Tathagata Das
发送时间：2019-09-17 16:13
主题：Re: how can I dynamic parse json in kafka when using Structured Streaming
收件人："lk_spar
You can use *from_json* built-in SQL function to parse json.
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html#from_json-org.apache.spark.sql.Column-org.apache.spark.sql.Column-
On Mon, Sep 16, 2019 at 7:39 PM lk_spark wrote:
> hi,all :
> I'm using Structured
hi,all :
I'm using Structured Streaming to read kafka , the data type is json String
, I want to parse it and conver to a datafrme , my code can't pass compile , I
don't know how to fix it:
val lines = messages.selectExpr("CAST(value AS STRING) as value").as[String]
val words = lines.map(
