[
https://issues.apache.org/jira/browse/HIVE-30129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30129:
----------------------------------
Labels: Usability (was: )
> Iceberg: HiveIcebergOutputFormat silently drops the cluster's
> parquet.compression override, writing Iceberg's own default codec instead
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30129
> URL: https://issues.apache.org/jira/browse/HIVE-30129
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: Usability
>
> HiveIcebergOutputFormat.setWriterLevelConfiguration is meant to bridge HS2's
> parquet.compression into Iceberg's write.parquet.compression-codec for any
> table that sets no codec of its own:
>
> {code}
>
>
> if (table.properties().get(TableProperties.PARQUET_COMPRESSION) == null &&
>
>
> jc.get(ParquetOutputFormat.COMPRESSION) != null) {
>
>
> table.properties().put(TableProperties.PARQUET_COMPRESSION,
> jc.get(ParquetOutputFormat.COMPRESSION));
>
> }
> {code}
>
>
>
>
>
> In practice jc.get(ParquetOutputFormat.COMPRESSION) is null in the actual
> Tez/LLAP write-task JobConf even when parquet.compression is set cluster-wide
> — it isn't a registered HiveConf.ConfVars, so nothing guarantees it's copied
> into a task's JobConf the way the ordinary (non-Iceberg) Parquet writer's
> config is. The bridge silently never fires, and the write falls through to
> Iceberg's own internal default (zstd since Iceberg 1.4.0), with no error and
> nothing visible in SHOW TBLPROPERTIES.
> *Impact*: any Iceberg table whose data is written fresh through Hive
> (CTAS/INSERT, not a metadata-only register of pre-existing files) and that
> doesn't set its own write.parquet.compression-codec, silently ignores the
> cluster-wide codec setting
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
