Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#7).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
with it now it's possible to run an Impala cluster without the Hive
Metastore, and without the Impala CatalogD. Impala Coordinators can
directly connect to an Iceberg REST server and fetch metadata for
databases and tables from there.
This was initially developed in the context of a company
Hackathon program, i.e. it was a team effort that I squashed
into a single commit and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support is in a preliminary state, which
means some details, e.g. configuration options may change in the
future. Currently, one can provide configuration for the REST
server via the following startup flags:
* --iceberg_catalog_name: Iceberg Catalog name
* --iceberg_rest_uri: Iceberg REST catalog uri
* --iceberg_rest_client_id: Iceberg REST catalog user ID
* --iceberg_rest_client_secret: Iceberg REST catalog user secret
* --iceberg_rest_warehouse_location: Iceberg REST catalog warehouse location
TODO: Instead of backend flags, we should use a config file.
Moreover, one needs to specify the following flags as well:
* --use_local_catalog=true
* --catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A
java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,427 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/7
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pro...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
