Dejan Krakovic created SPARK-60092:
--------------------------------------
Summary: Add FileType Catalyst data type with FILE EXTERNAL DDL
and JSON schema support
Key: SPARK-60092
URL: https://issues.apache.org/jira/browse/SPARK-60092
Project: Spark
Issue Type: Sub-task
Components: Spark Core, SQL
Affects Versions: 4.3.0
Reporter: Dejan Krakovic
Part 1 of the FILE type SPIP (SPARK-59132).
Add the FileType Catalyst data type, independent of parquet-java, so that
downstream consumers such as delta-spark, which represent column types as
Spark Catalyst DataTypes, have a type to refer to. There is currently no
FileType in OSS Spark.
Scope:
- FileType, a DataType that is physically a struct but does not extend
StructType, with SQL name "FILE EXTERNAL" and JSON name "file".
- The FILE keyword (non-reserved) and the FILE EXTERNAL DDL syntax. EXTERNAL
is required when declaring a FILE column.
- Schema parsing and serialization: DataType.fromJson / json and
DataType.fromDDL / sql round trip, including FILE nested in array, map
value and struct types.
- DataTypes.FileType for the Java API, and keyword documentation in
sql-ref-ansi-compliance.md.
Out of scope (later parts): physical representation (unsafe row, codegen,
encoders), casts, TO_FILE and related functions, comparability checks, the
Parquet schema converters, the UDF surface (FileRef) and Spark Connect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
