[
https://issues.apache.org/jira/browse/HIVE-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546007#comment-14546007
]
Alexander Pivovarov commented on HIVE-10711:
--------------------------------------------
1. Lets say noConditionalTaskThreshold is 3990MB and processMaxMemory = 4000MB
In this case noConditionalTaskThreshold adjustment will be skipped.
But if we slightly increase noConditionalTaskThreshold to lets say 4010 MB then
your code will work and will change noConditionalTaskThreshold to 2000 MB
20 MB change in noConditionalTaskThreshold makes significant change in final
value - 3990 MB vs 2000 MB (which is a big difference)
I think that noConditionalTaskThreshold adjustment should adjust
noConditionalTaskThreshold to smth which is close to 3990 MB
This is why I think using HIVEHASHTABLEMAXMEMORYUSAGE=0.9 instead of 0.5 makes
sense.
2. To improve the algo even more we can check if noConditionalTaskThreshold is
bigger than 90% of processMaxMemory and adjust it
{code}
if (noConditionalTaskThreshold / processMaxMemory >
HIVEHASHTABLEMAXMEMORYUSAGE) {
noConditionalTaskThreshold = processMaxMemory * HIVEHASHTABLEMAXMEMORYUSAGE
}
{code}
What you think?
> Tez HashTableLoader attempts to allocate more memory than available when
> HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD exceeds process max mem
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-10711
> URL: https://issues.apache.org/jira/browse/HIVE-10711
> Project: Hive
> Issue Type: Bug
> Reporter: Jason Dere
> Assignee: Jason Dere
> Attachments: HIVE-10711.1.patch, HIVE-10711.2.patch
>
>
> Tez HashTableLoader bases its memory allocation on
> HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD. If this value is largeer than the
> process max memory then this can result in the HashTableLoader trying to use
> more memory than available to the process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
