Hello Kurt Deschler, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22353
to look at the new patch set (#10).
Change subject: IMPALA-13586: Initial support for Iceberg REST Catalogs
......................................................................
IMPALA-13586: Initial support for Iceberg REST Catalogs
This patch adds initial support for Iceberg REST Catalogs. This means
now it's possible to run an Impala cluster without the Hive Metastore,
and without the Impala CatalogD. Impala Coordinators can directly
connect to an Iceberg REST server and fetch metadata for databases and
tables from there. The support is read-only, i.e. DDL and DML statements
are not supported yet.
This was initially developed in the context of a company Hackathon
program, i.e. it was a team effort that I squashed into a single commit
and polished the code a bit.
The Hackathon team members were:
* Daniel Becker
* Gabor Kaszab
* Kurt Deschler
* Peter Rozsa
* Zoltan Borok-Nagy
The Iceberg REST Catalog support can be configured via a Java properties
file, the location of it can be specified via:
--catalog_config_dirs: Directory of configuration files
Currently only one configuration file can be in the direcory as we only
support a single Catalog at a time. The following properties are mandatory
in the config file:
* connector.name=iceberg
* iceberg.catalog.type=rest
* iceberg.rest-catalog.uri
The first two properties can only be 'iceberg' and 'rest' for now, they
are needed for extensibility in the future.
Moreover, Impala Daemons need to specify the following flags to connect
to an Iceberg REST Catalog:
--use_local_catalog=true
--catalogd_deployed=false
Testing
* e2e added to test basic functionlity with against a custom-built
Iceberg REST server that delegates to HadoopCatalog under the hood
* Further testing, e.g. Ranger tests are expected in subsequent
commits
TODO:
* manual testing against Polaris / Lakekeeper, we could add automated
tests in a later patch
Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
---
M be/src/common/global-flags.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M bin/start-impala-cluster.py
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
A fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
A fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A java/iceberg-rest-catalog-test/pom.xml
A
java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M java/pom.xml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_rest.properties
A testdata/bin/run-iceberg-rest-server.sh
A testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-rest-catalog.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
A tests/custom_cluster/test_iceberg_rest_catalog.py
M www/catalog.tmpl
34 files changed, 1,515 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/22353/10
--
To view, visit http://gerrit.cloudera.org:8080/22353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1722b898b568d2f5689002f2b9bef59320cb088c
Gerrit-Change-Number: 22353
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
