[
https://issues.apache.org/jira/browse/HIVE-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14015878#comment-14015878
]
shanyu zhao commented on HIVE-7155:
-----------------------------------
[~thejas], Thank you for your feedback! I think it could be a legitimate case
when a cluster have larger mapreduce.map.memory.mb but smaller
templeton.mapper.memory.mb, for example, when the users are only use templeton
to submit jars. I think it is useful to separate the mapper's memory limit for
general mapreduce jobs and templeton's launch mapper.
In practice, the default 1GB memory limit for templeton's launch mapper is
sufficient for majority of job submission. But when we try to do hive query on
huge tables we'll hit memory limit for the templeton's launch mapper, because
hive tries to do quite a bit local processing. In that case, we want to
increase memory limit (to 2GB) for templeton's launch mapper, but not for
user's mapreduce jobs.
> WebHCat controller job exceeds container memory limit
> -----------------------------------------------------
>
> Key: HIVE-7155
> URL: https://issues.apache.org/jira/browse/HIVE-7155
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
> Affects Versions: 0.13.0
> Reporter: shanyu zhao
> Assignee: shanyu zhao
> Attachments: HIVE-7155.patch
>
>
> Submit a Hive query on a large table via WebHCat results in failure because
> the WebHCat controller job is killed by Yarn since it exceeds the memory
> limit (set by mapreduce.map.memory.mb, defaults to 1GB):
> {code}
> INSERT OVERWRITE TABLE Temp_InjusticeEvents_2014_03_01_00_00 SELECT * from
> Stage_InjusticeEvents where LogTimestamp > '2014-03-01 00:00:00' and
> LogTimestamp <= '2014-03-01 01:00:00';
> {code}
> We could increase mapreduce.map.memory.mb to solve this problem, but this way
> we are changing this setting system wise.
> We need to provide a WebHCat configuration to overwrite
> mapreduce.map.memory.mb when submitting the controller job.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
