[
https://issues.apache.org/jira/browse/HIVE-28450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17896395#comment-17896395
]
Sercan Tekin edited comment on HIVE-28450 at 11/7/24 4:39 PM:
--------------------------------------------------------------
Hi [~dengzh]! Thank for your answer.
I believe it is not about VM limits, this issue is related to how an array is
used as a transferable object.
Some VMs reserve some header words in an array, and when one (in this case
Hive) sets array/list size to *Integer.MAX_VALUE* (for instance
[this|https://github.com/apache/hive/blob/088f4d71a8c45b6f44d9f4be470f6eb7807a9ed8/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L1402-L1403]
property), the mentioned issue occurs.
To avoid this, we should be setting the transferable objects size slightly
lower than {*}Integer.MAX_VALUE{*}.
Java did it
[here|https://github.com/openjdk/jdk/blob/0e0dfca21f64ecfcb3e5ed7cdc2a173834faa509/src/java.base/share/classes/java/io/InputStream.java#L307-L313],
and Spark followed Java's approach
[here|https://github.com/apache/spark/blob/e5a5921968c84601ce005a7785bdd08c41a2d862/common/utils/src/main/scala/org/apache/spark/unsafe/array/ByteArrayUtils.java].
was (Author: JIRAUSER283532):
Hi [~dengzh]! Thank for your answer.
I believe it is not about VM limits, this issue is related to how an array is
used as a transferable object.
Some VMs reserve some header words in an array, and when one (in this case
Hive) sets array/list size to *Integer.MAX_VALUE* (for instance
[this|https://github.com/apache/hive/blob/088f4d71a8c45b6f44d9f4be470f6eb7807a9ed8/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L1402-L1403]
property), the mentioned issue occurs.
To avoid this, we should be setting the transferable objects size slightly
lower than {*}Integer.MAX_VALUE{*}.
Java does this
[here|https://github.com/openjdk/jdk/blob/0e0dfca21f64ecfcb3e5ed7cdc2a173834faa509/src/java.base/share/classes/java/io/InputStream.java#L307-L313],
and Spark follows Java's approach
[here|https://github.com/apache/spark/blob/e5a5921968c84601ce005a7785bdd08c41a2d862/common/utils/src/main/scala/org/apache/spark/unsafe/array/ByteArrayUtils.java].
> Follow the array size of JVM in Hive transferable objects
> ---------------------------------------------------------
>
> Key: HIVE-28450
> URL: https://issues.apache.org/jira/browse/HIVE-28450
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Affects Versions: 3.1.3, 4.0.0
> Reporter: Sercan Tekin
> Priority: Major
> Fix For: 4.1.0
>
>
> We are experiencing an issue with a partitioned table in Hive. When querying
> the table via the Hive CLI, the data retrieval works as expected without any
> errors. However, when attempting to query the same table through Spark, we
> encounter the following error in the HMS logs:
> {code:java}
> 2024-01-30 23:03:59,052 main DEBUG
> org.apache.logging.log4j.core.util.SystemClock does not support precise
> timestamps.
> Exception in thread "pool-7-thread-4" java.lang.OutOfMemoryError: Requested
> array size exceeds VM limit
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at
> org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:473)
> at
> org.apache.thrift.transport.TSaslServerTransport.write(TSaslServerTransport.java:42)
> at
> org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:227)
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.write(FieldSchema.java:517)
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.write(FieldSchema.java:456)
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.write(FieldSchema.java:394)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1423)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1250)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1116)
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1033)
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:890)
> at
> org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:786)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.write(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.write(ThriftHiveMetastore.java)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:58)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:603)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:600)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:600)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Exception in thread "pool-7-thread-6" java.lang.OutOfMemoryError: Requested
> array size exceeds VM limit
> Exception in thread "pool-7-thread-9" java.lang.OutOfMemoryError: Requested
> array size exceeds VM limit
> {code}
> This error appears to be related to the JVM’s conservative approach to array
> size allocation, which limits the maximum size of arrays to prevent
> OutOfMemoryError exceptions. For reference, you can see a similar
> implementation in the JVM source code here:
> https://github.com/openjdk/jdk/blob/0e0dfca21f64ecfcb3e5ed7cdc2a173834faa509/src/java.base/share/classes/java/io/InputStream.java#L307-L313
> Spark side implemented similar limit on their side, it would be good to
> implement the same thing on Hive side -
> https://github.com/apache/spark/blob/e5a5921968c84601ce005a7785bdd08c41a2d862/common/utils/src/main/scala/org/apache/spark/unsafe/array/ByteArrayUtils.java
> Workaround:
> As a temporary workaround, I have been able to mitigate the issue by setting
> the hive.metastore.batch.retrieve.table.partition.max configuration to a
> lower value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
