[
https://issues.apache.org/jira/browse/HIVE-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957156#comment-16957156
]
Jesus Camacho Rodriguez commented on HIVE-22238:
------------------------------------------------
Let's work on the more robust solution. It is not worth to trade a wrong
estimation for another, especially since overestimation will cause regressions
in mapjoin transformation. If we have identified that this is due to filter
conditions on the join keys accounted for more than once when selectivity is
computed, we can focus on that specific issue. At least for the
`getSelectivitySimpleTree` method (no binary operators below), it seems the fix
is not too complicated and will improve our estimates.
> PK/FK selectivity estimation underscales estimations
> ----------------------------------------------------
>
> Key: HIVE-22238
> URL: https://issues.apache.org/jira/browse/HIVE-22238
> Project: Hive
> Issue Type: Bug
> Components: Statistics
> Reporter: Zoltan Haindrich
> Assignee: Zoltan Haindrich
> Priority: Major
> Attachments: HIVE-22238.01.patch, HIVE-22238.02.patch,
> HIVE-22238.03.patch
>
>
> at [this
> point|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2182]
> the parent operators rownum is scaled according to pkfkselectivity
> however [pkfkselectivity is
> computed|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2157]
> on a whole subtree.
> Scaling it by that amount will count in estimation already used when
> parentstats was calculated...so depending on the number of upstream joins -
> this may lead to severe underestimations
> what happened was:
> * optimization was able to push the filter to the other side of the join
> * as a result the incoming data was already filtered
> * scaling down by the PK selectiviy - was actually already there...but a new
> "scaling" happened
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
