Sergey Shelukhin created HIVE-11587:
---------------------------------------
Summary: Fix memory estimates for mapjoin hashtable
Key: HIVE-11587
URL: https://issues.apache.org/jira/browse/HIVE-11587
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Wei Zheng
Due to the legacy in in-memory mapjoin and conservative planning, the memory
estimation code for mapjoin hashtable is currently not very good. It allocates
the probe erring on the side of more memory, not taking data into account
because unlike the probe, it's free to resize, so it's better for perf to
allocate big and hope for the best on data size. It is not true for hybrid case.
There's code to cap the initial allocation based on memory available (memSize
argument), but due to some code rot, the memory estimates from planning are not
even passed to hashtable anymore (there used to be two config settings,
hashjoin size fraction by itself, or hashjoin size fraction for group by case),
so it never caps the memory anymore below 1 Gb.
Initial capacity is estimated from input key count, and in hybrid join cache
can exceed Java memory due to number of segments.
There needs to be a review and fix of all this code.
Suggested improvements:
1) Make sure "initialCapacity" argument from Hybrid case is correct given the
number of segments. See how it's calculated from keys for regular case; it
needs to be adjusted accordingly for hybrid case if not done already.
2) Rename memSize to maxProbeSize, or something, make sure it's passed
correctly based on estimates that take into account both probe and data size,
esp. in hybrid case.
3) Cap single write buffer size to 8-16Mb.
4) For hybrid, don't pre-allocate WBs - only pre-allocate on write.
5) Change everywhere rounding up to power of two is used to rounding down, at
least for hybrid case (?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
