[
https://issues.apache.org/jira/browse/IMPALA-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125176#comment-18125176
]
ASF subversion and git services commented on IMPALA-15312:
----------------------------------------------------------
Commit 7e586ff1f57444ed4c89500061585d6933095ba8 in impala's branch
refs/heads/master from Aleksandr Efimov
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7e586ff1f ]
IMPALA-15312: Say when HBO changed a join's build side
IMPALA-15236 puts the provenance of an HBO cardinality next to the
number: the strategy that matched, the hash key, the estimate the planner
computed itself and the ratio between the two. That says where the number
came from, not whether it changed anything.
Join inversion is where the two come apart. Planner.invertJoins() weighs
the two inputs by cardinality, row size and parallelism, so a corrected
estimate can flip the verdict while the shape of the plan, its operator
counts and its node ids all stay put. On an outer-join-heavy report this
turned sixty joins from RIGHT OUTER back to LEFT OUTER, and comparing the
two plans by operator counts called them identical.
Run the same cost model a second time on the estimates the planner
computed for the two inputs, and record the outcome on the join when the
verdicts differ. At EXTENDED and above the join then prints
HBO chose the build side: without it this join would have been inverted
The comparison is local. A match further down the tree is already folded
into both numbers, so this answers "did the substitution on an input
decide this", not "would the plan differ under use_hbo_stats=false" - the
second question needs a second planning pass, which is more than a detail
line is worth.
Testing:
- New HboJoinInversionTest, three cases: no match leaves no note; a
seeded match that removes an inversion is recorded and printed; the
line stays out of the default explain level. The history is seeded
directly, so the tests do not depend on an earlier run.
Change-Id: Ifc9517936c3f93865917a5f3fa3924c641374fcd
Assisted-by: claude-opus-5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24781
Tested-by: Michael Smith <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
Reviewed-by: Steve Carlin <[email protected]>
> Show when an HBO cardinality changed a plan decision
> ----------------------------------------------------
>
> Key: IMPALA-15312
> URL: https://issues.apache.org/jira/browse/IMPALA-15312
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> h3. Problem
> IMPALA-15236 puts the provenance of an HBO cardinality next to the number:
> the canonicalization strategy that matched, the hash key, the estimate the
> planner had computed on its own, and the ratio between the two. That answers
> where the number came from. It does not answer whether the number changed the
> plan.
> The two questions come apart. Decisions that consume cardinalities are cost
> based, so a corrected estimate can flip one while leaving the shape of the
> plan alone. Join inversion is the clearest case: {{Planner.invertJoins()}}
> inverts a join when {{isInvertedJoinCheaper()}} says so, and that check
> weighs the two children by cardinality, average row size and node count.
> Correct the build side and the verdict can flip, so a join that was planned
> inverted goes back. The number of joins does not change, the
> broadcast/partitioned mix does not change, and the operator histogram of the
> plan does not change - only the side each join builds on.
> This is not hypothetical. On an outer-join-heavy query, HBO correcting a
> build-side aggregate from a few thousand rows down to fourteen turned sixty
> joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans
> by operator counts reported them as the same plan.
> Answering "what did HBO do to this plan" today means running the query twice,
> once with {{use_hbo_stats=false}}, and diffing the two plans. On a report
> that runs for minutes that is a measurement exercise rather than debugging.
> h3. Proposed change
> Mark the nodes where the HBO substitution changed a planner decision, not
> only the estimate, in the plan and in the profile.
> Two directions, each with a cost worth weighing:
> * Plan twice under a debug query option and diff the results. Exact, and only
> paid for when someone asks for it.
> * Carry the pre-HBO estimate through the cardinality computation and
> re-evaluate the cost based decisions with it. No second planning pass, but it
> has to propagate: a node whose own key did not match can still get a
> different cardinality because a descendant matched, and
> {{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
> reads only that field reports no change where there was one.
> h3. Note for the comparison itself
> Plan node ids do not move when a join flips: they are handed out as nodes are
> constructed, before {{invertJoins()}} runs, and {{invertJoin()}} keeps the
> id. Exchange ids do move - {{DistributedPlanner.createHashJoinFragment()}}
> numbers the left child's exchange before the right child's, so an inversion
> swaps the two, and the merge aggregation takes the next id from the same
> counter. A diff keyed on node id therefore reports differences that are only
> numbering; comparing the whole-plan operator histogram separates them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
