Max Gekk created SPARK-57815:
--------------------------------
Summary: Handle nanosecond-precision timestamp columns in
Hive-serde (STORED AS) tables
Key: SPARK-57815
URL: https://issues.apache.org/jira/browse/SPARK-57815
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond
precision).
h2. Problem
The Hive-serde path lacks a TIME-style guard:
{{HiveFileFormat.supportDataType}} returns false for {{TimeType}} (SPARK-57556)
but not for the nanosecond timestamp types; {{HiveInspectors.toInspector}} /
{{toTypeInfo}} have no nanosecond case; and the Hive ORC serde
(sql/hive/.../orc/OrcFileFormat.scala ~L227-258) silently downgrades nanosecond
timestamps to microsecond {{TimestampType}} on write. So {{CREATE TABLE ...
STORED AS ORC/PARQUET}} with a nanosecond column either loses precision
silently or fails with a generic legacy error.
h2. Goal
Give nanosecond timestamps in Hive-serde tables an intentional behavior -
either reject with a specific error (mirroring SPARK-57556 for TIME) or
explicitly support without precision loss; never a silent downgrade.
h2. Scope
Add nanosecond handling/rejection in {{HiveFileFormat.supportDataType}} and
{{HiveInspectors}}; decide reject-vs-support; add DDL tests ({{STORED AS
ORC/PARQUET}}).
h2. Acceptance criteria
* Creating/writing a Hive-serde table with a nanosecond column gives a clear
error or preserves precision - never a silent microsecond downgrade.
h2. Testing
{{HiveDDLSuite}} / {{HiveOrcSourceSuite}}.
h2. Dependencies
None - independent (Hive ORC datasource path resolved in SPARK-57455).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
