Hello Jason Fehr, Surya Hebbar, Daniel Vanko, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24118
to look at the new patch set (#5).
Change subject: IMPALA-14843: Show cancelled nodes in ExecSummary
......................................................................
IMPALA-14843: Show cancelled nodes in ExecSummary
In backend execution, a plan node instance could be closed before it
produces all its output rows, e.g., when a UnionNode reaches its output
limit, the children operands are all closed regardless what their states
(prepared, opened, executing, etc.) are. In the ExecSummary, such nodes
usually have 0 as the output cardinality without any indication, which
is confusing. This patch adds a "cancelled" marker in the Detail field
to represent such nodes, i.e., nodes that haven't reach the "Last Batch
Returned" state.
Implementation:
- Extends ExecSummaryDataPB and TExecStats to add a last_batch_returned
field to reflect such case.
- Adds a last_batch_returned_ flag in ExecNode to indicate whether the
"Last Batch Returned" event has been added for this node.
- Executor mirrors the flag to ExecSummaryDataPB for each node when
generating status report for coordinator. To lookup the ExecNode
instance based on the plan node id, a map is added in
FragmentInstanceState when the ExecNode tree is built.
- Coordinator appends the "cancelled" marker when returning
TExecSummary.
Example:
For query
"with l as
(select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
select STRAIGHT_JOIN count(*)
from (select * from tpch.lineitem a LIMIT 1) a
join (select * from l LIMIT 125000) b
on a.l_orderkey = -b.l_orderkey"
the ExecSummary is
Operator #Hosts #Inst Avg Time Max Time #Rows Est.
#Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------------------------
F01:ROOT 1 1 8.647us 8.647us
0 0
05:AGGREGATE 1 1 0.000ns 0.000ns 1
1 24.00 KB 16.00 KB FINALIZE
04:HASH JOIN 1 1 251.664us 251.664us 0
1 9.06 MB 4.75 MB INNER JOIN, BROADCAST
|--08:EXCHANGE 1 1 427.557us 427.557us 125.00K
125.00K 232.00 KB 337.52 KB UNPARTITIONED
| F05:EXCHANGE SENDER 1 1 3.007ms 3.007ms
54.05 KB 48.00 KB
| 07:EXCHANGE 1 1 499.833us 499.833us 125.00K
125.00K 5.12 MB 361.52 KB UNPARTITIONED
| F04:EXCHANGE SENDER 3 3 2.681ms 2.833ms
54.05 KB 48.00 KB
| 01:UNION 3 3 43.817us 48.670us 375.00K
125.00K 0 0
| |--03:SCAN HDFS 3 3 40.710us 49.726us 0
6.00M 0 264.00 MB tpch.lineitem (cancelled)
| 02:SCAN HDFS 3 3 5.813ms 7.886ms 377.86K
6.00M 48.23 MB 264.00 MB tpch.lineitem (cancelled)
06:EXCHANGE 1 1 10.288us 10.288us 1
1 16.00 KB 16.00 KB UNPARTITIONED
F00:EXCHANGE SENDER 3 3 32.298us 35.572us
31.00 B 48.00 KB
00:SCAN HDFS 3 3 76.692ms 85.439ms 3
1 48.07 MB 264.00 MB tpch.lineitem a
Note that "02:SCAN HDFS" and "03:SCAN HDFS" are cancelled due to
"01:UNION" reaches its limit.
Test:
- Added e2e test in query_test/test_observability.py
Generated-by: Claude Sonnet 4.5
Change-Id: Ibd1bf51112e7c8c5609bb9e9525d960efc524215
---
M be/src/exec/exec-node-util.h
M be/src/exec/exec-node.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/service/impala-http-handler.cc
M be/src/util/summary-util.cc
M be/src/util/summary-util.h
M common/protobuf/control_service.proto
M common/thrift/ExecStats.thrift
M tests/query_test/test_observability.py
13 files changed, 82 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24118/5
--
To view, visit http://gerrit.cloudera.org:8080/24118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibd1bf51112e7c8c5609bb9e9525d960efc524215
Gerrit-Change-Number: 24118
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Daniel Vanko <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Surya Hebbar <[email protected]>
