Hyukjin Kwon created SPARK-19849:
------------------------------------
Summary: Support ArrayType in to_json function/expression
Key: SPARK-19849
URL: https://issues.apache.org/jira/browse/SPARK-19849
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.2.0
Reporter: Hyukjin Kwon
After SPARK-19595, we could
{code}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
Seq(("""[{"a": 1}, {"a": 2}]""")).toDF("array").select(from_json(col("array"),
schema)).show()
{code}
Maybe, it'd be better if we provide a way to read it back via {{to_json}} as
below:
{code}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val schema = ArrayType(StructType(StructField("a", IntegerType) :: Nil))
val df = Seq("""[{"a":1}, {"a":2}]""").toDF("json").select(from_json($"json",
schema).as("array"))
df.show()
// Read back.
df.select(to_json($"array").as("json")).show()
{code}
{code}
+----------+
| array|
+----------+
|[[1], [2]]|
+----------+
+-----------------+
| json|
+-----------------+
|[{"a":1},{"a":2}]|
+-----------------+
{code}
Currently, it throws an exception as below:
{code}
org.apache.spark.sql.AnalysisException: cannot resolve 'structtojson(`array`)'
due to data type mismatch: structtojson requires that the expression is a
struct expression.;;
'Project [structtojson(array#30, Some(Asia/Seoul)) AS structtojson(array)#45]
+- Project
[jsontostruct(ArrayType(StructType(StructField(a,IntegerType,true)),true),
array#27, Some(Asia/Seoul)) AS array#30]
+- Project [value#25 AS array#27]
+- LocalRelation [value#25]
at
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:80)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:72)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
