[
https://issues.apache.org/jira/browse/IMPALA-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13656 started by Noemi Pap-Takacs.
-------------------------------------------------
> MERGE redundantly accumulates memory in HDFS WRITER
> ---------------------------------------------------
>
> Key: IMPALA-13656
> URL: https://issues.apache.org/jira/browse/IMPALA-13656
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Noemi Pap-Takacs
> Assignee: Noemi Pap-Takacs
> Priority: Major
> Fix For: Impala 4.5.0
>
>
> When we want to merge Iceberg tables that have lots of partitions, the
> execution will use much more memory than needed, possibly resulting in a
> Memory Limit Exceeded error.
> It happens because when IcebergMergeImpl creates the insert table sink it
> doesn't set 'inputIsClustered' to true. Therefore HdfsTableSink expects
> random input and keeps the output writers open for every partition, which
> results in high memory consumption.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
