[
https://issues.apache.org/jira/browse/HIVE-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680740#comment-14680740
]
Gopal V commented on HIVE-11502:
--------------------------------
A custom hashcode can be used internal to Hive (i.e group-by etc), but not
externally to hive (bucketing into HDFS, results of hash() functions).
Because that would break external assumptions in a non-backwards-compatible way.
The reason shuffle + merge is more uniform is because it starts using [murmur
hashes|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java#L366]
for UNIFORM trait RS instead of the builtin writable hash funcs (which are
skewed).
You will probably notice that using a vectorized input format like ORC would
not have the issue you're hitting, since the vector transform inside the
operator pipeline gives hive the opportunity to use per-operator specific
optimizations.
> Map side aggregation is extremely slow
> --------------------------------------
>
> Key: HIVE-11502
> URL: https://issues.apache.org/jira/browse/HIVE-11502
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Physical Optimizer
> Affects Versions: 1.2.0
> Reporter: Yongzhi Chen
> Assignee: Yongzhi Chen
>
> For the query as following:
> {noformat}
> create table tbl2 as
> select col1, max(col2) as col2
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000) and
> it is in type double, the map side aggregation is very slow. I ran the query
> which took more than 3 hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set hive.map.aggr = false;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
