[
https://issues.apache.org/jira/browse/HIVE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-25975:
------------------------------
Description:
The first version of the ClusteredWriter in Hive-Iceberg will be lenient for
bucketed tables: i.e. the records do not need to be ordered by the bucket
values, the writer will just close its current file and open a new one for
out-of-order records.
This is suboptimal for the long-term due to creating many small files. Spark
uses a UDF to compute the bucket value for each record and therefore it is able
to order the records by bucket values, achieving optimal clustering.
> Optimize ClusteredWriter for bucketed Iceberg tables
> ----------------------------------------------------
>
> Key: HIVE-25975
> URL: https://issues.apache.org/jira/browse/HIVE-25975
> Project: Hive
> Issue Type: Improvement
> Reporter: Ádám Szita
> Assignee: Ádám Szita
> Priority: Major
>
> The first version of the ClusteredWriter in Hive-Iceberg will be lenient for
> bucketed tables: i.e. the records do not need to be ordered by the bucket
> values, the writer will just close its current file and open a new one for
> out-of-order records.
> This is suboptimal for the long-term due to creating many small files. Spark
> uses a UDF to compute the bucket value for each record and therefore it is
> able to order the records by bucket values, achieving optimal clustering.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
