Aleksandar Trajković created SPARK-58876:
--------------------------------------------
Summary: Oracle: Read DATE and TIMESTAMP as TimestampNTZType
Key: SPARK-58876
URL: https://issues.apache.org/jira/browse/SPARK-58876
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Aleksandar Trajković
Oracle {{DATE}} and plain {{TIMESTAMP}} are zoneless, and both arrive from the
Oracle JDBC driver under JDBC type code {{Types.TIMESTAMP}}. Today
{{OracleDialect}}
has no mapping for {{Types.TIMESTAMP}}, so they fall through to the shared
default
and are read as {{TimestampType}} (or {{TimestampNTZType}} only when the JDBC
read
option {{preferTimestampNTZ}} is set).
Mapping a zoneless source type to {{TimestampType}} -- an instant rendered in
the
session time zone -- is not faithful: the same wall-clock value is
reinterpreted as
an instant and shifts under a non-UTC {{spark.sql.session.timeZone}}.
{{TimestampNTZType}} (timestamp without time zone) represents these Oracle types
correctly.
This maps Oracle {{DATE}} and plain {{TIMESTAMP}} to {{TimestampNTZType}} by
default,
independent of {{preferTimestampNTZ}}. Oracle {{TIMESTAMP WITH TIME ZONE}} and
{{TIMESTAMP WITH LOCAL TIME ZONE}} carry an instant, arrive under their own JDBC
codes, and are unaffected.
This mirrors the sibling JDBC dialect timestamp-mapping corrections in Spark
4.0:
SPARK-47871 (Oracle, write side), SPARK-48387 (Postgres), SPARK-47515 (MySQL).
As a behavior change it is gated by a legacy config
{{spark.sql.legacy.oracle.timestampNTZMapping.enabled}} (default false) that
restores
the previous behavior, and is documented in the SQL migration guide.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
