Hello Daniel Becker, Kurt Deschler, Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22367
to look at the new patch set (#11).
Change subject: IMPALA-13654: Tolerate missing data files of Iceberg tables
......................................................................
IMPALA-13654: Tolerate missing data files of Iceberg tables
Before this patch we got a TableLoadingException for missing data files.
This means the IcebergTable will be in an incomplete state in Impala's
memory, therefore we won't be able to do any operation on it.
We should continue table loading in such cases, and only throw exception
for queries that are about to read the missing data files.
This way ROLLBACK / DROP PARTITION, and some SELECT statements should
still work.
If Impala is running in strict mode via CatalogD flag
--iceberg_allow_datafiles_in_table_location_only, and an Iceberg table
has data files outside of table location, we still raise an exception
and leave the table in an unloaded state. To retain this behavior, the
IOException we threw is substituted to TableLoadingException which fits
better to logic errors anyway.
Testing
* added e2e tests
Change-Id: If753619d8ee1b30f018e90157ff7bdbe5d7f1525
---
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-missing-data-files.test
M tests/query_test/test_iceberg.py
6 files changed, 268 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/22367/11
--
To view, visit http://gerrit.cloudera.org:8080/22367
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If753619d8ee1b30f018e90157ff7bdbe5d7f1525
Gerrit-Change-Number: 22367
Gerrit-PatchSet: 11
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
