[
https://issues.apache.org/jira/browse/SPARK-57815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57815:
-----------------------------
Shepherd: Max Gekk
> Handle nanosecond-precision timestamp columns in Hive-serde (STORED AS) tables
> ------------------------------------------------------------------------------
>
> Key: SPARK-57815
> URL: https://issues.apache.org/jira/browse/SPARK-57815
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
> precision).
> h2. Problem
> The Hive-serde path lacks a TIME-style guard:
> {{HiveFileFormat.supportDataType}} returns false for {{TimeType}}
> (SPARK-57556) but not for the nanosecond timestamp types;
> {{HiveInspectors.toInspector}} / {{toTypeInfo}} have no nanosecond case; and
> the Hive ORC serde (sql/hive/.../orc/OrcFileFormat.scala ~L227-258) silently
> downgrades nanosecond timestamps to microsecond {{TimestampType}} on write.
> So {{CREATE TABLE ... STORED AS ORC/PARQUET}} with a nanosecond column either
> loses precision silently or fails with a generic legacy error.
> h2. Goal
> Give nanosecond timestamps in Hive-serde tables an intentional behavior -
> either reject with a specific error (mirroring SPARK-57556 for TIME) or
> explicitly support without precision loss; never a silent downgrade.
> h2. Scope
> Add nanosecond handling/rejection in {{HiveFileFormat.supportDataType}} and
> {{HiveInspectors}}; decide reject-vs-support; add DDL tests ({{STORED AS
> ORC/PARQUET}}).
> h2. Acceptance criteria
> * Creating/writing a Hive-serde table with a nanosecond column gives a clear
> error or preserves precision - never a silent microsecond downgrade.
> h2. Testing
> {{HiveDDLSuite}} / {{HiveOrcSourceSuite}}.
> h2. Dependencies
> None - independent (Hive ORC datasource path resolved in SPARK-57455).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
