[
https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967105#action_12967105
]
Namit Jain commented on HIVE-1830:
----------------------------------
After HIVE-1642, joins are automatically converted into map-joins at physical
optimization time.
However, this may lead to problems.
For eg: consider the query:
select T1.val, count(1) from T1 join T2 on T1.key=T2.key group by T1.val
This will have 2 map-reduce jobs, one for the join and the other for group by.
Before HIVE-1642, the partial group for aggregation will be performed in the
reducer where the join is performed.
However, after HIVE-1642, the same will be performed in the mapper. The local
task will confirm that there is just
enough memory to hold the map-join data. Hoever, it does not take into account
the memory needed for partial group
by.
So, in case there is group by followed by join, it is a good idea to reduce the
memory given to the local task to validate
if there is enough memory to fit small table - it can be controlled by a new
configuration paramter, but it can be some
default: say 70% of total memory (instead of 90%).
Also, the group by may still run out of memory, so it might be a good idea to
check in group by for free memory and
periodically flush memory
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
