[
https://issues.apache.org/jira/browse/IMPALA-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110430#comment-18110430
]
ASF subversion and git services commented on IMPALA-15236:
----------------------------------------------------------
Commit da61c7a407f1348ef177359be9dc33b1b3f3ccc4 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=da61c7a40 ]
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
At EXTENDED, a scan whose estimate came from a recorded run reads:
00:SCAN HDFS [functional.alltypes, RANDOM]
partition predicates: `year` = CAST(2009 AS INT)
HDFS partitions=12/24 files=12 size=238.68KB
predicates: int_col = CAST(1 AS INT), string_col = '1'
stored statistics:
table: rows=7.30K size=478.45KB
partitions: 12/12 rows=3.65K
columns: all
extrapolated-rows=disabled max-scan-range-rows=622
mem-estimate=80.00MB mem-reservation=32.00KB thread-reservation=1
tuple-ids=0 row-size=17B cardinality=365 (from HBO)
HBO match: strategy=EXPR_REWRITE, hash=56704b19611ac2584658f691edebdf22,
original estimate=115, ratio=3.17
in pipelines: 00(GETNEXT)
The planner had 115 rows for that scan on its own, and the run it
matched had 365.
The line is text rather than fields on TPlanNode next to the hbo_hash_keys
that already travel there. Those keys are written when store_hbo_stats is
on, to record a run; this says what a lookup found while planning, and it
is read where the estimate itself is read - in the plan, in the profile,
and in whatever a person pastes into a ticket. A structured form of it is
worth having the day something other than a reader wants it.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, key ignores partition constants)". An EXPR_REWRITE match
keeps its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The caveat says what the key leaves out, not what the node does, because
the key covers the subtree: an aggregate over a scan on partition
predicates carries it while having no partition predicates of its own,
which is what the goldens here show.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24693
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Quanlong Huang <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Expose HBO matching provenance in query profiles
> ------------------------------------------------
>
> Key: IMPALA-15236
> URL: https://issues.apache.org/jira/browse/IMPALA-15236
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> h3. Problem
> When HBO replaces a plan node's estimated cardinality, the plan and profile
> currently annotate it only as "(from HBO)". This does not show which
> canonicalization strategy matched or how the HBO value differs from the
> original estimate. As a result, surprising estimates are difficult to
> investigate and profile tools such as plan-graph.py cannot explain why
> history was reused.
> h3. Proposed change
> Expose match provenance when an HBO cardinality is applied:
> * Show the selected canonicalization strategy, for example "(from HBO
> EXPR_REWRITE)".
> * Preserve the original planner estimate alongside the applied HBO
> cardinality.
> * Consider exposing the opaque HBO hash key used for the match, similar to
> TupleCacheNode.
> The descriptive canonicalized key and raw query material should not be
> exposed. Non-HBO estimates should remain unchanged. This task does not define
> matching rules for future CPU, memory, or runtime statistics and does not
> require stabilizing the V2 profile format.
> h3. Acceptance criteria
> * Text plan/profile output identifies the selected strategy for an HBO hit.
> * The original estimate remains available when HBO replaces it.
> * Tests cover an HBO hit and a non-HBO path.
> * If a key is exposed, only the opaque hash is shown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
