Aleksandr Efimov created IMPALA-15361:
-----------------------------------------
Summary: TIMESTAMP column stats are written to a Hive 4 metastore
and never read back
Key: IMPALA-15361
URL: https://issues.apache.org/jira/browse/IMPALA-15361
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Aleksandr Efimov
Impala 5.0.0-SNAPSHOT (build e8d42f753d), Apache Hive 4.0.0 metastore.
{{COMPUTE STATS}} on a 10-column table left a row in {{TAB_COL_STATS}} for
every column, TIMESTAMP ones included, each with {{NUM_DISTINCTS}} set. On the
next {{REFRESH}}, catalogd logged {{Failed to load column stats ... incompatible
with column type TIMESTAMP}} for the four TIMESTAMP columns and nothing for the
other six, and {{SHOW COLUMN STATS}} has shown -1 for those four ever since.
Recomputing does not change it. In 78 minutes of catalogd log there are 212
such warnings on 58 tables, every one TIMESTAMP.
That warning fires only when a {{ColumnStatisticsObj}} did arrive and
{{ColumnStats.update()}} returned false, so the metastore is answering with
something {{isSetLongStats()}} rejects -- while the write side sets
{{longStats}} at {{ColumnStats.java:896-897}}. The comment on the read branch
states the assumption Hive 4 appears to break:
{code}
case TIMESTAMP: // Hive and Impala use LongColumnStatsData for timestamps.
isCompatible = statsData.isSetLongStats(); // ColumnStats.java:610-611
{code}
What I have not read is the wire: which member of the union comes back.
{{timestampStats}} is the candidate, since {{TimestampColumnStatsData}} appears
nowhere in Impala's sources while DATE has its own arm -- but a missing read
arm and HMS 4 converting on write want different fixes, and I have not tried
HMS 3.x.
Meanwhile {{COMPUTE STATS}} reports success and planning falls back to defaults
on every TIMESTAMP predicate and join key. One trade worth naming before a fix
picks a side: writing {{timestampStats}} too would take stats away from older
readers on the same metastore.
Filed under IMPALA-13647.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
