[
https://issues.apache.org/jira/browse/HIVE-27984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Butao Zhang updated HIVE-27984:
-------------------------------
Description:
We had some discussions about column stats thrift api isssue before:
[https://www.mail-archive.com/dev@hive.apache.org/msg144294.html] ,
and i also did some attempt to fix this issue from TrinoDB side when i playing
TrinoDB&Hive4 [https://github.com/trinodb/trino-hive-apache/pull/43] .
Recently, i tried to use starrocks([https://github.com/StarRocks/starrocks])
which uses a hms client compatible with Hive3 to query Hive4, and i also
encountered this issue([https://github.com/StarRocks/starrocks/issues/38620]),
{code:java}
Caused by: org.apache.thrift.TApplicationException: Required field 'engine' is
unset! Struct:TableStatsRequest(dbName:testdb, tblName:testsr, colNames:[id],
engine:null)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[libthrift-0.13.0.jar:0.13.0]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:4161)
~[hive-apache-3.1.2-13.jar:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:4148)
~[hive-apache-3.1.2-13.jar:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:735)
~[starrocks-fe.jar:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:727)
~[starrocks-fe.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_221]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_221]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_221]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
~[hive-apache-3.1.2-13.jar:?]
at com.sun.proxy.$Proxy32.getTableColumnStatistics(Unknown Source)
~[?:?]
{code}
In addition, i also found similar ticket in Impala
https://issues.apache.org/jira/browse/IMPALA-12539 .
That is to say, if other components(Impala&TrinoDB&Starrocks) want to use hive3
hms client to get the column stats to make CBO, they will get the execption
from Hive4. I think it is necessary to support backward compatibility to make
other compontents work well with hive4(hms4).
was:
We had some discussions about column stats thrift api isssue before:
[https://www.mail-archive.com/dev@hive.apache.org/msg144294.html] ,
and i also did some attempt to fix this issue from TrinoDB side when i playing
TrinoDB&Hive4 [https://github.com/trinodb/trino-hive-apache/pull/43] .
Recently, i tried to use starrocks([https://github.com/StarRocks/starrocks])
which uses a hms client compatible with Hive3 to query Hive4, and i also
encountered this issue,
{code:java}
Caused by: org.apache.thrift.TApplicationException: Required field 'engine' is
unset! Struct:TableStatsRequest(dbName:testdb, tblName:testsr, colNames:[id],
engine:null)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
~[libthrift-0.13.0.jar:0.13.0]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:4161)
~[hive-apache-3.1.2-13.jar:?]
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:4148)
~[hive-apache-3.1.2-13.jar:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:735)
~[starrocks-fe.jar:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:727)
~[starrocks-fe.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_221]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_221]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_221]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
~[hive-apache-3.1.2-13.jar:?]
at com.sun.proxy.$Proxy32.getTableColumnStatistics(Unknown Source)
~[?:?]
{code}
In addition, i also found similar ticket in Impala
https://issues.apache.org/jira/browse/IMPALA-12539 .
That is to say, if other components(Impala&TrinoDB&Starrocks) want to use hive3
hms client to get the column stats to make CBO, they will get the execption
from Hive4. I think it is necessary to support backward compatibility to make
other compontents work well with hive4(hms4).
> Support backward compatibility of hms thrift struct about column stats
> ----------------------------------------------------------------------
>
> Key: HIVE-27984
> URL: https://issues.apache.org/jira/browse/HIVE-27984
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Reporter: Butao Zhang
> Assignee: Butao Zhang
> Priority: Major
>
> We had some discussions about column stats thrift api isssue before:
> [https://www.mail-archive.com/dev@hive.apache.org/msg144294.html] ,
> and i also did some attempt to fix this issue from TrinoDB side when i
> playing TrinoDB&Hive4 [https://github.com/trinodb/trino-hive-apache/pull/43] .
> Recently, i tried to use starrocks([https://github.com/StarRocks/starrocks])
> which uses a hms client compatible with Hive3 to query Hive4, and i also
> encountered this
> issue([https://github.com/StarRocks/starrocks/issues/38620]),
>
> {code:java}
> Caused by: org.apache.thrift.TApplicationException: Required field 'engine'
> is unset! Struct:TableStatsRequest(dbName:testdb, tblName:testsr,
> colNames:[id], engine:null)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> ~[libthrift-0.13.0.jar:0.13.0]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:4161)
> ~[hive-apache-3.1.2-13.jar:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:4148)
> ~[hive-apache-3.1.2-13.jar:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:735)
> ~[starrocks-fe.jar:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:727)
> ~[starrocks-fe.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_221]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_221]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_221]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_221]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
> ~[hive-apache-3.1.2-13.jar:?]
> at com.sun.proxy.$Proxy32.getTableColumnStatistics(Unknown Source)
> ~[?:?]
> {code}
>
> In addition, i also found similar ticket in Impala
> https://issues.apache.org/jira/browse/IMPALA-12539 .
> That is to say, if other components(Impala&TrinoDB&Starrocks) want to use
> hive3 hms client to get the column stats to make CBO, they will get the
> execption from Hive4. I think it is necessary to support backward
> compatibility to make other compontents work well with hive4(hms4).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
