[
https://issues.apache.org/jira/browse/IMPALA-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17947770#comment-17947770
]
ASF subversion and git services commented on IMPALA-13268:
----------------------------------------------------------
Commit 3c24706c72818a1668159a428d4f2afcadea9f27 in impala's branch
refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3c24706c7 ]
IMPALA-13268: Integrate Iceberg ScanMetrics into Impala query profiles
When calling planFiles() on an Iceberg table, it can give us some
metrics like total planning time, number of data/delete files and
manifests, how many of these could be skipped etc.
This change integrates these metrics into the query profile, under the
"Frontend" section. These metrics are per-table, so if multiple tables
are scanned for the query there will be multiple sections in the
profile.
Note that we only have these metrics for a table if Iceberg needs to be
used for planning for that table, e.g. if a predicate is pushed down to
Iceberg or if there is time travel. For tables where Iceberg was not
used in planning, the profile will contain a short note describing this.
To facilitate pairing the metrics with scans, the metrics header
references the plan node responsible for the scan. This will always be
the top level node for the scan, so it can be a SCAN node, a JOIN node
or a UNION node depending on whether the table has delete files.
Testing:
- added EE tests in iceberg-scan-metrics.tests
- added a test in PlannerTest.java that asserts on the number of
metrics; if it changes in a new Iceberg release, the test will fail
and we can update our reporting
Change-Id: I080ee8eafc459dad4d21356ac9042b72d0570219
Reviewed-on: http://gerrit.cloudera.org:8080/22501
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Daniel Becker <[email protected]>
> Use Iceberg ScanMetrics in the query profiles
> ---------------------------------------------
>
> Key: IMPALA-13268
> URL: https://issues.apache.org/jira/browse/IMPALA-13268
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Gabor Kaszab
> Assignee: Daniel Becker
> Priority: Major
> Labels: impala-iceberg
>
> When calling planFiles() on Iceberg, it can give us some metrics like number
> of files, manifests, total planning time, etc. see the list here:
> [https://github.com/apache/iceberg/blob/806da5cfc7dba7b8fd872cf7fc6a6b36ac8a3876/core/src/main/java/org/apache/iceberg/metrics/ScanMetrics.java#L27]
> We could add these metrics into our query profile.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
