[
https://issues.apache.org/jira/browse/HIVE-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17897236#comment-17897236
]
Sercan Tekin commented on HIVE-6179:
------------------------------------
I was able to reproduce this issue when authentication is enabled. So, I
suspect this is due to some VMs reserve some header words in an array.
Please see the solution on JDK side
https://github.com/openjdk/jdk/blob/0e0dfca21f64ecfcb3e5ed7cdc2a173834faa509/src/java.base/share/classes/java/io/InputStream.java#L307-L313
And, the solution on Spark side which follows the same approach
https://github.com/apache/spark/commit/e5a5921968c84601ce005a7785bdd08c41a2d862#diff-607488c104788f0156de87abab394cf33aa76148b1e3d122d328e165a25c1838R22
> OOM occurs when query spans to a large number of partitions
> -----------------------------------------------------------
>
> Key: HIVE-6179
> URL: https://issues.apache.org/jira/browse/HIVE-6179
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.12.0
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Priority: Major
>
> When executing a query against a large number of partitions, such as "select
> count(\*) from table", OOM error may occur because Hive fetches the metadata
> for all partitions involved and tries to store it in memory.
> {code}
> 2014-01-09 13:14:17,090 ERROR metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invoke(141)) - java.lang.OutOfMemoryError: Java heap
> space
> at java.util.Arrays.copyOf(Arrays.java:2367)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
> at java.lang.StringBuffer.append(StringBuffer.java:237)
> at
> org.apache.derby.impl.sql.conn.GenericStatementContext.appendErrorInfo(Unknown
> Source)
> at
> org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
> at
> org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:191)
> at
> org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:379)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.loopJoinOrderedResult(MetaStoreDirectSql.java:641)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:410)
> at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:205)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1433)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1420)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:122)
> at com.sun.proxy.$Proxy7.getPartitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:2128)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
> {code}
> The above error happened when executing "select count(\*)" on a table with
> 40K partitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
