Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22367 )
Change subject: IMPALA-13654: Tolerate missing data files of Iceberg tables ...................................................................... IMPALA-13654: Tolerate missing data files of Iceberg tables Before this patch we got a TableLoadingException for missing data files. This means the IcebergTable will be in an incomplete state in Impala's memory, therefore we won't be able to do any operation on it. We should continue table loading in such cases, and only throw exception for queries that are about to read the missing data files. This way ROLLBACK / DROP PARTITION, and some SELECT statements should still work. If Impala is running in strict mode via CatalogD flag --iceberg_allow_datafiles_in_table_location_only, and an Iceberg table has data files outside of table location, we still raise an exception and leave the table in an unloaded state. To retain this behavior, the IOException we threw is substituted to TableLoadingException which fits better to logic errors anyway. Testing * added e2e tests Change-Id: If753619d8ee1b30f018e90157ff7bdbe5d7f1525 Reviewed-on: http://gerrit.cloudera.org:8080/22367 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-missing-data-files.test M tests/query_test/test_iceberg.py 6 files changed, 269 insertions(+), 29 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/22367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If753619d8ee1b30f018e90157ff7bdbe5d7f1525 Gerrit-Change-Number: 22367 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
