[
https://issues.apache.org/jira/browse/HIVE-25843?focusedWorklogId=705976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-705976
]
ASF GitHub Bot logged work on HIVE-25843:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/22 10:49
Start Date: 10/Jan/22 10:49
Worklog Time Spent: 10m
Work Description: marton-bod commented on pull request #2917:
URL: https://github.com/apache/hive/pull/2917#issuecomment-1008749526
@pvary when working with metadata queries, the IO can be part of the Task,
e.g.:
https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/DataFilesTable.java#L139
If we remove the hadoop config from this IO to make the splits smaller, then
we wouldn't be able to inject back the config later on the Tez worker-side,
since the Task API does not provide any options for that.
So I've disabled the serialization skipping for metadata queries. We can
look into making this work for metadata queries as well in the future if the
need arises, but I think that might require some Iceberg-side API changes too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 705976)
Time Spent: 40m (was: 0.5h)
> Add flag to disable Iceberg FileIO config serialization
> -------------------------------------------------------
>
> Key: HIVE-25843
> URL: https://issues.apache.org/jira/browse/HIVE-25843
> Project: Hive
> Issue Type: Improvement
> Reporter: Marton Bod
> Assignee: Marton Bod
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Hive serializes the Iceberg table object into each individual split. Since
> the FileIO is part of the Iceberg table and it has its own hadoop
> configuration, this configuration will be the dominant factor determining the
> size of the serialized split. In our tests we have found that due to this
> serialized config, iceberg splits are 15-20x larger than normal Hive splits
> (which led to OOM in some of our perf tests).
> This PR proposes to introduce a config which can turn off this config
> serialization, and let the deserializer-side fill out the config values
> instead (which works for Hive executors, since they have all the config
> values in hand). This can reduce the Iceberg split size by ~20x based on
> local tests.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
