[
https://issues.apache.org/jira/browse/SPARK-61132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61132:
-----------------------------------
Labels: pull-request-available (was: )
> Raise a clear error for TIMESTAMP_NTZ columns in the Hive ORC data source
> -------------------------------------------------------------------------
>
> Key: SPARK-61132
> URL: https://issues.apache.org/jira/browse/SPARK-61132
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Anshul Baliga
> Priority: Major
> Labels: pull-request-available
>
> The Hive ORC data source (\{{spark.sql.orc.impl=hive}}) has no mapping for
> \{{TIMESTAMP_NTZ}}. \{{OrcSerializer.toHiveCompatibleDataType}} rewrites only
> the nanosecond timestamp types and collated strings, so a \{{TIMESTAMP_NTZ}}
> column reaches Hive's type parser as the string \{{timestamp_ntz}} and the
> write fails with a Hive internal error that does not say which column is
> wrong or why.
> SPARK-60545 fixed the same problem for Hive serde tables by rejecting
> \{{TIMESTAMP_NTZ}} in \{{HiveFileFormat.supportDataType}} with
> \{{UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE}}, but the sibling
> \{{OrcFileFormat.supportDataType}} in \{{sql/hive}} was not updated. The
> three earlier rejections in that method (\{{TIME}} in SPARK-51590, the
> nanosecond timestamps in SPARK-57166, the geospatial types in SPARK-60519)
> were applied to both methods.
> *Steps to reproduce (verified on master):*
> {code:sql}
> SET spark.sql.orc.impl=hive;
> CREATE TABLE t (ts TIMESTAMP_NTZ) USING ORC; -- succeeds
> INSERT INTO t VALUES (TIMESTAMP_NTZ'1970-01-01 00:00:01'); -- fails inside
> Hive
> {code}
> *Expected:* \{{CREATE TABLE}} fails with
> \{{[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support
> the column `ts` of the type "TIMESTAMP_NTZ".}}, like the Hive serde format
> and like the other unsupported types in Hive ORC.
> *Proposed fix:* add \{{case _: TimestampNTZType => false}} to
> \{{org.apache.spark.sql.hive.orc.OrcFileFormat.supportDataType}}, with a test
> in \{{HiveOrcQuerySuite}} covering the write path (top-level and nested), the
> read path with a user-specified schema, and \{{CREATE TABLE ... USING ORC}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
