Asif created SPARK-50840: ---------------------------- Summary: TimestampType wrongly gets mapped to TimestampNTZ when reloading metadata from Hive, if timestamp alias is set to NTZ Key: SPARK-50840 URL: https://issues.apache.org/jira/browse/SPARK-50840 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.5.4, 4.0.0 Reporter: Asif
1) Hive does not support spark's TimeStampNTZType 2) the mapping of the default spark's LTZ timestamp is incorrect, which is a separate and much bigger issue. If we create a hive table with timestamp field such that timestamp alias ( defined by property spark.sql.timestampType) is pointing to TimestampTypes.TIMESTAMP_LTZ, the metadata in Hive will be "timestamp". Later if we change the Timestamp alias to TimestampTypes.TIMESTAMP_NTZ, and do any operation, such that table metadata is reloaded from Hive, the timestamp type generated in Spark would get value of TimestampTypes.TIMESTAMP_NTZ, instead of TimestampType ( i.e LTZ). This would eventually result in parsing exception in Hive layer as it does not recognize timestamp NTZ. Will be opening a PR, with bug test -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org