Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21760 )
Change subject: IMPALA-12349: Support Apache Hive 2.x in Impala ...................................................................... IMPALA-12349: Support Apache Hive 2.x in Impala Like IMPALA-10871, this patch adds MetastoreShim to support Apache Hive 2.x. At the build time, based on the environment variable IMPALA_HIVE_DIST_TYPE one of the three shims is added to as source using the fe/pom.xml build plugin. And select the dependencies related to Hive in the fe/pom.xml based on the environment variable IMPALA_HIVE_MAJOR_VERSION. There are some duplicate classes under compat-apache-hive2 directory, e.g. fe/src/compat-apache-hive-2/java/ org/apache/impala/catalog/events/MetastoreEvents.java duplicates fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java The class in compat-apache-hive2 is a simplified version that works with Apache Hive 2.x. So we don't need to extract lots of Hive-dependent codes in MetastoreEvents.java into the metastore shim. Due to this, the build process simply remove the original source code when building on Apache Hive 2. Additionally, it should be noted that all the code in the fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive directory comes from Apache Hive 3.x, original source: https://github.com/apache/hive/blob/branch-3.1 In order to reduce the unnecessary intrusion into the code, skip all tests when building with Apache Hive 2.x. If wanting to build Impala adapted to Apache Hive 2.x, please set the following environment variables before `source bin/impala-config.sh`: export USE_APACHE_COMPONENTS=true export USE_APACHE_HIVE_2=true TODO: - IMPALA-14581: Support testing related to Apache Hive 2 in the minicluster. Testing: - Compile using the -package option to obtain the package. After deployment, perform all types of query tests, including SELECT, INSERT, CREATE TABLE, ALTER TABLE, COMPUTE STATUS, etc. In addition, comprehensive testing has been conducted on the metadata auto-synchronization functionality. The tests confirm that all event types are supported except for AlterDatabaseEvent, AllocWriteIdEvent, AbortTxnEvent, PseudoAbortTxnEvent, and CommitCompactionEvent. It is worth noting that these unsupported events are not generated in Apache Hive 2, so their lack of processing support does not impact the functionality. Change-Id: Ib5f104dc8d131835b8118b9d54077471db65681c Reviewed-on: http://gerrit.cloudera.org:8080/21760 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M bin/rat_exclude_files.txt M buildall.sh M common/function-registry/CMakeLists.txt M fe/pom.xml A fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java A fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/common/ValidTxnWriteIdList.java A fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java A fe/src/compat-apache-hive-2/java/org/apache/hadoop/hive/metastore/api/TableValidWriteIds.java A fe/src/compat-apache-hive-2/java/org/apache/impala/catalog/Hive2MetastoreShimBase.java A fe/src/compat-apache-hive-2/java/org/apache/impala/catalog/events/MetastoreEvents.java A fe/src/compat-apache-hive-2/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java A fe/src/compat-apache-hive-2/java/org/apache/impala/compat/MetastoreShim.java A fe/src/compat-apache-hive-2/java/org/apache/thrift/TConfiguration.java A fe/src/compat-apache-hive-2/java/org/apache/thrift/transport/TEndpointTransport.java M fe/src/main/java/org/apache/impala/analysis/ConvertTableToIcebergStmt.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Hive3MetastoreShimBase.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/HiveMetadataFormatUtils.java M java/calcite-planner/pom.xml M java/impala-package/pom.xml M java/pom.xml M java/shaded-deps/hive-exec/pom.xml A testdata/bin/patch_hive2.sh A testdata/cluster/hive2/patch0-HIVE-18755.diff M tests/common/environ.py 36 files changed, 8,179 insertions(+), 138 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib5f104dc8d131835b8118b9d54077471db65681c Gerrit-Change-Number: 21760 Gerrit-PatchSet: 123 Gerrit-Owner: ttttttz <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Pranav Lodha <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Zihao Ye <[email protected]> Gerrit-Reviewer: ttttttz <[email protected]>
