[
https://issues.apache.org/jira/browse/HIVE-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887496#comment-15887496
]
Lefty Leverenz edited comment on HIVE-12492 at 9/18/17 5:19 AM:
----------------------------------------------------------------
Doc note: This adds *hive.auto.convert.join.hashtable.max.entries* to
HiveConf.java, so it needs to be documented in the wiki.
* [Configuration Properties -- Query and DDL Execution |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]
Added a TODOC2.2 label.
Edit (7/Mar/17): HIVE-16137 changes the default value to 40,000,000 so that's
what should be documented in the wiki.
Typo alert: In the parameter description, "does not take affect" should be
"does not take effect." This can be corrected in the wiki.
Edit (18/Sep/17): HIVE-17276 fixes the typo in the parameter description and
HIVE-17521 changes the default value to 21,000,000 (both in release 3.0.0).
was (Author: le...@hortonworks.com):
Doc note: This adds *hive.auto.convert.join.hashtable.max.entries* to
HiveConf.java, so it needs to be documented in the wiki.
* [Configuration Properties -- Query and DDL Execution |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]
Added a TODOC2.2 label.
Edit (7/Mar/17): HIVE-16137 changes the default value to 40,000,000 so that's
what should be documented in the wiki.
Typo alert: In the parameter description, "does not take affect" should be
"does not take effect." This can be corrected in the wiki.
> MapJoin: 4 million unique integers seems to be a probe plateau
> --------------------------------------------------------------
>
> Key: HIVE-12492
> URL: https://issues.apache.org/jira/browse/HIVE-12492
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
> Affects Versions: 1.3.0, 1.2.1, 2.0.0
> Reporter: Gopal V
> Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-12492.01.patch, HIVE-12492.02.patch,
> HIVE-12492.patch
>
>
> After 4 million keys, the map-join implementation seems to suffer from a
> performance degradation.
> The hashtable build & probe time makes this very inefficient, even if the
> data is very compact (i.e 2 ints).
> Falling back onto the shuffle join or bucket map-join is useful after 2^22
> items.
> (Note: this fixes a statsutil issue - due to the extra clone() in the column
> stats path)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
