[
https://issues.apache.org/jira/browse/HIVE-30020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Ahuja updated HIVE-30020:
--------------------------------
Attachment: 0001-HIVE-30020-Fix-TIMESTAMP-WITH-LOCAL-TIME-ZONE-data-l.patch
> TIMESTAMP WITH LOCAL TIME ZONE columns lose data / fail to read on ORC tables
> ------------------------------------------------------------------------------
>
> Key: HIVE-30020
> URL: https://issues.apache.org/jira/browse/HIVE-30020
> Project: Hive
> Issue Type: Bug
> Reporter: Vikram Ahuja
> Assignee: Vikram Ahuja
> Priority: Major
> Attachments:
> 0001-HIVE-30020-Fix-TIMESTAMP-WITH-LOCAL-TIME-ZONE-data-l.patch
>
>
> TIMESTAMP WITH LOCAL TIME ZONE is not properly supported in ORC format.
> Though we are able to create and insert data. Select query throws
> UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT exception. The
> ORC read/write code (org.apache.hadoop.hive.ql.io.orc) never had a case added
> for this type when it was introduced, which produces two distinct failures:
> 1. Silent data loss on write. WriterImpl.setColumn() switches on
> PrimitiveCategory to populate a TimestampColumnVector, but has no case
> TIMESTAMPLOCALTZ and no default branch so the assignment is silently skipped,
> leaving the column at its zero-initialized value. INSERT completes with no
> error, but every row's timestamp is written as epoch 0.
> 2. Read failure. Reading such a column back out of ORC (e.g. via SELECT,
> using Hive's FetchTask local-read optimization) throws, because:
> - OrcStruct.createObjectInspector(int columnId, List<OrcProto.Type> types)
> has no case for TIMESTAMP_INSTANT (the ORC on-disk type for this Hive type)
> UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT.
> - RecordReaderImpl.nextValue() has no case for TIMESTAMP_INSTANT →
> IllegalArgumentException: Unknown type timestamp with local time zone.
>
>
> Steps to reproduce:
> {code:java}
> CREATE TABLE timestampltz_formats (
> formatid string,
> tsval timestamp with local time zone
> )
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe';
> LOAD DATA LOCAL INPATH '.../data/files/timestamps_mixed_formats.txt'
> overwrite into table timestampltz_formats;
> CREATE TABLE timestampltz_orc_format (
> formatid string,
> tsval timestamp with local time zone
> )
> stored as orc;
> insert into timestampltz_orc_format select * from timestampltz_formats;
> SELECT * FROM timestampltz_orc_format;
> {code}
>
> Observed: SELECT throws UnsupportedOperationException: Unknown type
> TIMESTAMP_INSTANT. If the read-side gap is patched in isolation, the
> exception disappears but every tsval comes back as epoch 0 (e.g. 1969-12-31
> 16:00:00.0 US/Pacific) instead of the inserted value, because the write
> silently dropped the data.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
