[
https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-5611:
----------------------------
Description:
We discover a performance issue with savepoint when the metadata table is
enabled. It is due to unnecessary scanning of the metadata table when the
number of partitions is large. When the metadata table is enabled, in the
savepoint operation, for each partition, the metadata table is scanned, which
leads to a lot of S3 requests. The solution is to batch the list calls of all
partitions (HUDI-5485).
We need to revisit
> Revisit metadata-table-based file listing calls and use batch lookup instead
> ----------------------------------------------------------------------------
>
> Key: HUDI-5611
> URL: https://issues.apache.org/jira/browse/HUDI-5611
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Priority: Major
>
> We discover a performance issue with savepoint when the metadata table is
> enabled. It is due to unnecessary scanning of the metadata table when the
> number of partitions is large. When the metadata table is enabled, in the
> savepoint operation, for each partition, the metadata table is scanned, which
> leads to a lot of S3 requests. The solution is to batch the list calls of
> all partitions (HUDI-5485).
>
> We need to revisit
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
