[
https://issues.apache.org/jira/browse/HIVE-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Ahuja updated HIVE-27505:
--------------------------------
Description: Hive.getAllPartitionsOf() tries to list partitions in one go,
it can cause memory related issues with wide tables(>2000 columns) with large
amount of partitions when data is being transferred from HMS to HS2 using
Thrift. Optimise the flow to use PartitionIterable so that partitions can be
fetched in batches rather than one go. (was: Hive.getAllPartitionsOf() tries
to list partitions in one go, it can cause issues with wide tables(>2000
columns) with large amount of partitions. Optimise the flow to use
PartitionIterable so that partitions can be fetched in batches rather than one
go.)
> Optimize the size of partition object from HMS in various flows
> ---------------------------------------------------------------
>
> Key: HIVE-27505
> URL: https://issues.apache.org/jira/browse/HIVE-27505
> Project: Hive
> Issue Type: Bug
> Reporter: Vikram Ahuja
> Assignee: Vikram Ahuja
> Priority: Major
> Labels: pull-request-available
>
> Hive.getAllPartitionsOf() tries to list partitions in one go, it can cause
> memory related issues with wide tables(>2000 columns) with large amount of
> partitions when data is being transferred from HMS to HS2 using Thrift.
> Optimise the flow to use PartitionIterable so that partitions can be fetched
> in batches rather than one go.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
