Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/22501 )
Change subject: IMPALA-13268: Integrate Iceberg ScanMetrics into Impala query profiles ...................................................................... IMPALA-13268: Integrate Iceberg ScanMetrics into Impala query profiles When calling planFiles() on Iceberg, it can give us some metrics like total planning time, number of data/delete files and manifests, how many of these could be skipped etc. This change integrates these metrics into the query profile, under the "Frontend" section. These metrics are per-table, so if multiple tables are scanned for the query there will be multiple sections in the profile. Note that we only have these metrics for a table if Iceberg needs to be used for planning for that table, e.g. if a predicate is pushed down to Iceberg or if there is time travel. For tables where Iceberg was not used in planning, the profile will contain a short note describing this. To facilitate pairing the metrics with scans, the metrics header references the plan node responsible for the scan. This will always be the top level node for the scan, so it can be a SCAN node, a JOIN node or a UNION node depending on whether the table has delete files. Testing: - added EE tests in iceberg-scan-metrics.tests - added a test in PlannerTest.java that asserts on the number of metrics; if it changes in a new Iceberg release, the test will fail and we can update our reporting Change-Id: I080ee8eafc459dad4d21356ac9042b72d0570219 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/FrontendProfile.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-scan-metrics.test M tests/query_test/test_iceberg.py 8 files changed, 365 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/22501/6 -- To view, visit http://gerrit.cloudera.org:8080/22501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I080ee8eafc459dad4d21356ac9042b72d0570219 Gerrit-Change-Number: 22501 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Gabor Kaszab <gaborkas...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Noemi Pap-Takacs <npaptak...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
