Do your schema inference and then apply the JSON schema using withColumn
overwriting the String representation
From: Nirav Patel
Date: Tuesday, October 2, 2018 at 5:00 PM
To:
Cc: spark users
Subject: Re: CSV parser - how to parse column containing json data
I need to inferSchema from
I need to inferSchema from CSV as well. As per your solution, I am creating
SructType only for Json field. So how am I going to mix and match here?
i.e. do type inference for all fields but json field and use custom
json_schema for json field.
On Thu, Aug 30, 2018 at 5:29 PM Brandon Geise
wro
If you know your json schema you can create a struct and then apply that using
from_json:
val json_schema = StructType(Array(StructField(“x”, StringType, true),
StructField(“y”, StringType, true), StructField(“z”, IntegerType, true)))
.withColumn("_c3", from_json(col("_c3_signals"),json_s
