[
https://issues.apache.org/jira/browse/HIVE-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663199#comment-14663199
]
Yongzhi Chen commented on HIVE-11502:
-------------------------------------
It is a regression, I ran the query in 0.13.1 version with hive.map.aggr is
true, it finished in 20 seconds.
In the master branch, the major time spend in following stack.
{noformat}
HashMap<K,V>.getEntry(Object) line: 465
HashMap<K,V>.get(Object) line: 417
PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(String) line: 373
PrimitiveTypeInfo.getPrimitiveTypeEntry() line: 85
PrimitiveTypeInfo.getPrimitiveCategory() line: 63
WritableDoubleObjectInspector(AbstractPrimitiveObjectInspector).getPrimitiveCategory()
line: 58
ObjectInspectorUtils.compare(Object, ObjectInspector, Object, ObjectInspector,
MapEqualComparer) line: 694
ObjectInspectorUtils.compare(Object, ObjectInspector, Object, ObjectInspector)
line: 668
ListObjectsEqualComparer$FieldComparer.areEqual(Object, Object) line: 127
ListObjectsEqualComparer.areEqual(Object[], Object[]) line: 172
KeyWrapperFactory$ListKeyWrapper.equals(Object) line: 101
HashMap<K,V>.getEntry(Object) line: 467
HashMap<K,V>.get(Object) line: 417
GroupByOperator.processHashAggr(Object, ObjectInspector, KeyWrapper) line: 777
GroupByOperator.processKey(Object, ObjectInspector) line: 693
GroupByOperator.process(Object, int) line: 761
SelectOperator(Operator<T>).forward(Object, ObjectInspector) line: 837
SelectOperator.process(Object, int) line: 88
TableScanOperator(Operator<T>).forward(Object, ObjectInspector) line: 837
TableScanOperator.process(Object, int) line: 97
MapOperator$MapOpCtx.forward(Object) line: 162
MapOperator.process(Writable) line: 508
ExecMapper.map(Object, Object, OutputCollector, Reporter) line: 163
{noformat}
It seems that heavily used
PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(String) slows down the
query. So I change the code to store the PrimitiveTypeEntry as instance
variable in PrimitiveTypeInfo. This does improve the performance a lot, now
the query can finish in 1 hour. But it is still very slow.
I checked 0.13.1 code, it uses Hashmap too, but much much faster.
I do not know why the HashMap search is so slow in master branch(and 1.1 or
later version).
> Map side aggregation is extremely slow
> --------------------------------------
>
> Key: HIVE-11502
> URL: https://issues.apache.org/jira/browse/HIVE-11502
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, Physical Optimizer
> Affects Versions: 1.2.0
> Reporter: Yongzhi Chen
> Assignee: Yongzhi Chen
>
> For the query as following:
> {noformat}
> create table tbl2 as
> select col1, max(col2) as col2
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000),
> the map side aggregation is very slow. I ran the query which took more than 3
> hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set hive.map.aggr = false;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
