[
https://issues.apache.org/jira/browse/HIVE-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794155#comment-17794155
]
Ayush Saxena commented on HIVE-27520:
-------------------------------------
I am able to reproduce this on master.
{noformat}
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:705)
at java.util.HashMap.putVal(HashMap.java:630)
at java.util.HashMap.put(HashMap.java:613)
at
org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.put(HashMapWrapper.java:110)
at
org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.putRow(HashMapWrapper.java:134)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.loadTableContainerData(MapJoinTestConfig.java:781)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoin(MapJoinTestConfig.java:846)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoinImplementation(MapJoinTestConfig.java:988)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoinImplementation(MapJoinTestConfig.java:971)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.executeTestImplementation(TestMapJoinOperator.java:1969)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.executeRowModeHashMap(TestMapJoinOperator.java:1895)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.doExecuteTest(TestMapJoinOperator.java:1854)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.executeTestInnerBigOnly(TestMapJoinOperator.java:1813)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.executeTest(TestMapJoinOperator.java:1787)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.doTestMultiKey3(TestMapJoinOperator.java:1225)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator.testMultiKey3(TestMapJoinOperator.java:1158)
{noformat}
This is not a regression, the code is there since 2014. It is a configuration
which defines the intial size of the HashMap, if you put an insane number, it
is on you and we can't define at what number we will get OOM, it would vary as
per env.
Reducing the priority from critical
> Out of Memory when set hive.hashtable.initialCapacity to a large number
> -----------------------------------------------------------------------
>
> Key: HIVE-27520
> URL: https://issues.apache.org/jira/browse/HIVE-27520
> Project: Hive
> Issue Type: Bug
> Reporter: ConfX
> Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When setting {{hive.hashtable.initialCapacity}} to a large number, Hive
> throws an out-of-memory exception due to inappropriate checking and handling.
> *There is no checking or error-handling logic for this parameter at all.*
> h2. StackTrace:
> {noformat}
> java.lang.IllegalArgumentException: Illegal initial capacity: -923553553
>
>
> at java.base/java.util.HashMap.<init>(HashMap.java:445)
>
>
> at
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.<init>(HashMapWrapper.java:89)
> at
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper.<init>(HashMapWrapper.java:81)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoin(MapJoinTestConfig.java:837)
>
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoinImplementation(MapJoinTestConfig.java:988)
> at
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.MapJoinTestConfig.createMapJoinImplementation(MapJoinTestConfig.java:971)
> {noformat}
> h2. Reproduce:
> (1) Set {{hive.hashtable.initialCapacity}} to a large value, e.g., 1223962862
> (2) Run a simple test that exercises this parameter, e.g.
> {{org.apache.hadoop.hive.ql.exec.vector.mapjoin.TestMapJoinOperator#testMultiKey3}}
>
> For an easy reproduction, run the reproduce.sh in the attachment.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
