Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23113 )
Change subject: IMPALA-13267: Display number of partitions for Iceberg tables ...................................................................... IMPALA-13267: Display number of partitions for Iceberg tables Before this change, query plans and profile reported only a single partition even for partitioned Iceberg tables, which was misleading for users. Now we can display the number of scanned partitions correctly for both partitioned and unpartitioned Iceberg tables. This is achieved by extracting the partition values from the file descriptors and storing them in the IcebergContentFileStore. Instead of storing this information redundantly in all file descriptors, we store them in one place and reference the partition metadata in the FDs with an id. This also gives the opportunity to optimize memory consumption in the Catalog and Coordinator as well as reduce network traffic between them in the future. Time travel is handled similarly to oldFileDescMap. In that case we don't know the total number of partitions in the old snapshot, so the output is [Num scanned partitions]/unknown. Testing: - Planner tests - E2E tests - partition transforms - partition evolution - DROP PARTITION - time travel Change-Id: Ifb2f654bc6c9bdf9cfafc27b38b5ca2f7b6b4872 Reviewed-on: http://gerrit.cloudera.org:8080/23113 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M common/fbs/IcebergObjects.fbs M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-merge-insert-only.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-merge.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-mixed-file-format.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-predicates-disabled-subsetting.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-predicates.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-delete.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-hash-join.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-resources.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-update.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample-iceberg.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-partitions.test M tests/query_test/test_iceberg.py 26 files changed, 735 insertions(+), 403 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/23113 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifb2f654bc6c9bdf9cfafc27b38b5ca2f7b6b4872 Gerrit-Change-Number: 23113 Gerrit-PatchSet: 15 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
