[
https://issues.apache.org/jira/browse/IMPALA-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951691#comment-17951691
]
ASF subversion and git services commented on IMPALA-14071:
----------------------------------------------------------
Commit ed6c19cf0ccef0d55bd9397245f61abc3327e439 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ed6c19cf0 ]
IMPALA-14071: Refactor helper methods around cardinality bounding
There are multiple ways to do cardinality multipication that also avoid
integer overflow. Some helper methods available are:
- MathUtil.saturatingMultiplyCardinalities()
- PlanNode.checkedMultiply()
- LongMath.saturatedMultiply()
This patch intent to simplify things by:
- MathUtil.saturatingMultiplyCardinalities() with
PlanNode.checkedMultiply() into MathUtil.multiplyCardinalities().
- MathUtil.saturatingAddCardinalities() with
PlanNode.checkedAdd() into MathUtil.addCardinalities().
- Move PlanNode.smallestValidCardinality() to MathUtil.
- Make MathUtil.saturatingMultiply() and MathUtil.saturatingAdd() simply
a wrapper for LongMath.saturatedMultiply() and LongMath.saturatedAdd()
accordingly.
multiplyCardinalities(), addCardinalities(), and
smallestValidCardinality() have cardinality Preconditions check.
Harden cardinality calculation in several places by using
multiplyCardinalities() and addCardinalities() accordingly. Added sanity
check PlanNode.verifyCardinality() that is evaluated at the end of
PlanNode.computeStats(). This ensure that cardinality_ and
inputCardinality_ is always valid after PlanNode.computeStats().
Also fixed bug in ExchangeNode.estimateTotalQueueByteSize() that prevent
calculation against negative cardinality or non-positive num nodes.
Testing:
Pass Following FE and EE tests:
CardinalityTest
MathUtilTest
PlannerTest#testSpillableBufferSizing+testResourceRequirements
TpcdsCpuCostPlannerTest
TpcdsPlannerTest
metadata/test_explain.py
Change-Id: I505ab11cfa1024feb4ceac4cffe9c3283be228ce
Reviewed-on: http://gerrit.cloudera.org:8080/22897
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Replace MathUtil.saturatingMultiplyCardinalities() with
> PlanNode.checkedMultiply()
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-14071
> URL: https://issues.apache.org/jira/browse/IMPALA-14071
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Minor
> Fix For: Impala 5.0.0
>
>
> The two methods has been used interchangeably in FE code.
> Ideally, we should stick with one for cardinality calculation purpose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
