[
https://issues.apache.org/jira/browse/HIVE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906497#comment-13906497
]
Sergey Shelukhin commented on HIVE-6429:
----------------------------------------
Measured memory savings; 1 million row hashtable; option one - double+int key,
double+string value.
Before (not sure why there's one less key, might be heap dump or profiler
glitch)
|Class|Objects|Shallow Size|Retained Size|
|java.lang.Object[]|2999997|79999920|256383712|
|org.apache.hadoop.hive.serde2.io.DoubleWritable|2999997|71999928|71999928|
|byte[]|999999|32383968|32383968|
|java.util.HashMap$Entry|999999|31999968|352383616|
|java.util.ArrayList|999999|23999976|208383760|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer|999999|23999976|232383736|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList|999999|23999976|160383808|
|org.apache.hadoop.io.LongWritable|999999|23999976|23999976|
|org.apache.hadoop.io.Text|999999|23999976|56383912|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinKey|999999|15999984|87999912|
|java.util.HashMap$Entry[]|1|8388624|360772272|
|java.util.HashMap|1|48|360772320|
After
|Class|Objects|Shallow Size|Retained Size|
|byte[]|2000000|72384000|72384000|
|java.lang.Object[]|2000000|56000000|184384000|
|org.apache.hadoop.hive.serde2.io.DoubleWritable|2000000|48000000|48000000|
|java.util.HashMap$Entry|1000000|32000000|320384000|
|java.util.ArrayList|1000000|24000000|208384000|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer|1000000|24000000|232384000|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList|1000000|24000000|160384000|
|org.apache.hadoop.io.Text|1000000|24000000|56384000|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyBytes|1000000|16000000|56000000|
|java.util.HashMap$Entry[]|1|8388624|328772016|
|java.util.HashMap|1|48|328772064|
Savings are ~9%.
Not as large as expected, as apparently Java has large overhead for byte
arrays, 9-byte array takes 32 bytes and 17-byte one has 40.
Still, reasonably good, perhaps the patch can be extended.
Or different hashtable used...
> MapJoinKey has large memory overhead in typical cases
> -----------------------------------------------------
>
> Key: HIVE-6429
> URL: https://issues.apache.org/jira/browse/HIVE-6429
> Project: Hive
> Issue Type: Improvement
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
> Attachments: HIVE-6429.01.patch, HIVE-6429.WIP.patch, HIVE-6429.patch
>
>
> The only thing that MJK really needs it hashCode and equals (well, and
> construction), so there's no need to have array of writables in there.
> Assuming all the keys for a table have the same structure, for the common
> case where keys are primitive types, we can store something like a byte array
> combination of keys to reduce the memory usage. Will probably speed up
> compares too.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
