Hello Jason Fehr, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22818
to look at the new patch set (#2).
Change subject: IMPALA-13931: TestIcebergRestCatalog.test_rest_catalog_basic
failed at setup
......................................................................
IMPALA-13931: TestIcebergRestCatalog.test_rest_catalog_basic failed at setup
There were several issues with test_rest_catalog_basic which made it
fail in environments that used Ozone or S3.
Missing dependency of Ozone and S3 classes:
* This is resolved in iceberg-rest-catalog-test/pom.xml by adding
a dependency to impala-executor-deps
Hadoop configuration was initialized properly:
* run-iceberg-rest-server.sh used Maven to run Iceberg REST Catalog in
which case Maven is in charge of setting the CLASSPATH but the
core-site/ozone-site/etc. config files were not on it, so the
REST Catalog used a default Hadoop configuration that wasn't good
for our environment.
* To overcome the CLASSPATH problem now we create a runnable JAR in
iceberg-rest-catalog-test/pom.xml and also generate the proper
CLASSPATH during compilation.
* run-iceberg-rest-server.sh now uses java -cp to run the REST
Catalog
S3 builds threw NoSuchMethodException for the "create" method of
ApacheHttpClientConfigurations:
* The Iceberg library dynamically load its http client builders
to workaround an error, see details in
https://github.com/apache/iceberg/issues/6715
* So the Iceberg lib dynamically wants to load the "create" method
of its own ApacheHttpClientConfigurations class but it fails
with NoSuchMethodException.
* The critical code is invoked from Impala's IcebergMetadataScanner's
ScanMetadataTable() method which happens to be invoked through
JNI from the C++ backend.
* The context class loader of such threads are NULL, which means
Java will use the bootstrap class loader to load classes and methods,
but that doesn't have the proper resources on its classpath.
* To overcome this issue we set the context class loader for the thread
to the class loader that originally loaded the IcebergMetadataScanner
class.
Change-Id: I9dc0e30aeaff0b8de41426ba38506383b4af472c
---
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalog.java
M fe/src/main/java/org/apache/impala/common/JniUtil.java
M fe/src/main/java/org/apache/impala/util/IcebergMetadataScanner.java
M java/iceberg-rest-catalog-test/pom.xml
M
java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M testdata/bin/run-iceberg-rest-server.sh
6 files changed, 79 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/22818/2
--
To view, visit http://gerrit.cloudera.org:8080/22818
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9dc0e30aeaff0b8de41426ba38506383b4af472c
Gerrit-Change-Number: 22818
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
