Vikram Ahuja created HIVE-30020:
-----------------------------------
Summary: TIMESTAMP WITH LOCAL TIME ZONE columns lose data / fail
to read on ORC tables
Key: HIVE-30020
URL: https://issues.apache.org/jira/browse/HIVE-30020
Project: Hive
Issue Type: Bug
Reporter: Vikram Ahuja
Assignee: Vikram Ahuja
TIMESTAMP WITH LOCAL TIME ZONE is not properly supported in ORC format. Though
we are able to create and insert data. Select query throws
UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT exception. The
ORC read/write code (org.apache.hadoop.hive.ql.io.orc) never had a case added
for this type when it was introduced, which produces two distinct failures:
1. Silent data loss on write. WriterImpl.setColumn() switches on
PrimitiveCategory to populate a TimestampColumnVector, but has no case
TIMESTAMPLOCALTZ and no default branch so the assignment is silently skipped,
leaving the column at its zero-initialized value. INSERT completes with no
error, but every row's timestamp is written as epoch 0.
2. Read failure. Reading such a column back out of ORC (e.g. via SELECT,
using Hive's FetchTask local-read optimization) throws, because:
- OrcStruct.createObjectInspector(int columnId, List<OrcProto.Type> types)
has no case for TIMESTAMP_INSTANT (the ORC on-disk type for this Hive type)
UnsupportedOperationException: Unknown type TIMESTAMP_INSTANT.
- RecordReaderImpl.nextValue() has no case for TIMESTAMP_INSTANT →
IllegalArgumentException: Unknown type timestamp with local time zone.
Steps to reproduce:
{code:java}
CREATE TABLE timestampltz_formats (
formatid string,
tsval timestamp with local time zone
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe';
LOAD DATA LOCAL INPATH '.../data/files/timestamps_mixed_formats.txt'
overwrite into table timestampltz_formats;
CREATE TABLE timestampltz_orc_format (
formatid string,
tsval timestamp with local time zone
)
stored as orc;
insert into timestampltz_orc_format select * from timestampltz_formats;
SELECT * FROM timestampltz_orc_format;
{code}
Observed: SELECT throws UnsupportedOperationException: Unknown type
TIMESTAMP_INSTANT. If the read-side gap is patched in isolation, the exception
disappears but every tsval comes back as epoch 0 (e.g. 1969-12-31 16:00:00.0
US/Pacific) instead of the inserted value, because the write silently dropped
the data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
