Hello Quanlong Huang, Zihao Ye, Pranav Lodha, Michael Smith, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21760
to look at the new patch set (#116).
Change subject: IMPALA-12349: Support Apache Hive 2.x in Impala
......................................................................
IMPALA-12349: Support Apache Hive 2.x in Impala
Like IMPALA-10871, this patch adds MetastoreShim to support Apache
Hive 2.x. At the build time, based on the environment variable
IMPALA_HIVE_DIST_TYPE one of the three shims is added to as source
using the fe/pom.xml build plugin. And select the dependencies
related to Hive in the fe/pom.xml based on the environment variable
IMPALA_HIVE_MAJOR_VERSION. There are some duplicate classes under
compat-apache-hive2 directory, e.g. fe/src/compat-apache-hive-2/java/
org/apache/impala/catalog/events/MetastoreEvents.java duplicates
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
The class in compat-apache-hive2 is a simplified version that works
with Apache Hive 2.x. So we don't need to extract lots of Hive-dependent
codes in MetastoreEvents.java into the metastore shim. Due to this,
the build process simply remove the original source code when building
on Apache Hive 2. Additionally, it should be noted that all
the code in the fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive
directory comes from Apache Hive 3.x, original
source: https://github.com/apache/hive/blob/branch-3.1
In order to reduce the unnecessary intrusion into the code, skip all
tests when building with Apache Hive 2.x. If wanting to build Impala
adapted to Apache Hive 2.x, please set the following environment variables
before `source bin/impala-config.sh`:
export USE_APACHE_COMPONENTS=true
export USE_APACHE_HIVE_2=true
TODO:
- IMPALA-14581: Support testing related to Apache Hive 2 in the minicluster.
Testing:
- Compile using the -package option to obtain the package. After
deployment, perform all types of query tests, including SELECT, INSERT,
CREATE TABLE, ALTER TABLE, COMPUTE STATUS, etc. In addition, comprehensive
testing has been conducted on the metadata auto-synchronization functionality.
The tests confirm that all event types are supported except for
AlterDatabaseEvent,
AllocWriteIdEvent, AbortTxnEvent, PseudoAbortTxnEvent, and
CommitCompactionEvent.
It is worth noting that these unsupported events are not generated in Apache
Hive 2,
so their lack of processing support does not impact the functionality.
Change-Id: Ib5f104dc8d131835b8118b9d54077471db65681c
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/jenkins/build-all-flag-combinations.sh
M bin/rat_exclude_files.txt
M buildall.sh
M common/function-registry/CMakeLists.txt
M fe/pom.xml
A
fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
A
fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/common/ValidTxnWriteIdList.java
A
fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java
A
fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/metastore/api/TableValidWriteIds.java
A
fe/src/compat-apache-hive-2/java/org/apache/impala/catalog/Hive2MetastoreShimBase.java
A
fe/src/compat-apache-hive-2/java/org/apache/impala/catalog/events/MetastoreEvents.java
A
fe/src/compat-apache-hive-2/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
A fe/src/compat-apache-hive-2/java/org/apache/impala/compat/MetastoreShim.java
A fe/src/compat-apache-hive-2/java/org/apache/thrift/TConfiguration.java
A
fe/src/compat-apache-hive-2/java/org/apache/thrift/transport/TEndpointTransport.java
M fe/src/main/java/org/apache/impala/analysis/ConvertTableToIcebergStmt.java
M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/HiveMetadataFormatUtils.java
M java/calcite-planner/pom.xml
M java/pom.xml
A testdata/bin/patch_hive2.sh
A testdata/cluster/hive2/patch0-HIVE-18755.diff
M tests/common/environ.py
34 files changed, 8,080 insertions(+), 127 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21760/116
--
To view, visit http://gerrit.cloudera.org:8080/21760
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5f104dc8d131835b8118b9d54077471db65681c
Gerrit-Change-Number: 21760
Gerrit-PatchSet: 116
Gerrit-Owner: ttttttz <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Pranav Lodha <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Zihao Ye <[email protected]>
Gerrit-Reviewer: ttttttz <[email protected]>
