Hi Fokko, That makes sense, thank you for the suggestion! The issue was quite severe for us that we had to fork the repo and have a fix ourselves in order to run PyIceberg without our applications going OOM. So I think there will be value in getting the proposed config property out as early as possible for the larger community.
I'm still on the fence regarding 17.0.0 upgrade. There are clear functional upsides, but I feel that constraining PyIceberg to just one published version would make the adoption of PyIceberg difficult for our users. Users writing new applications won't have trouble with it, but users intending to use PyIceberg in an existing application may have to upgrade their PyArrow versions which could be a deterrent (or a welcome nudge). Would it be worth starting that discussion on a separate thread? Sung On 2024/08/02 17:57:17 Fokko Driesprong wrote: > Hey Sung, > > Typically we only push patches into the minor versions, we could also go to > version 0.8.0 immediately. > > Regarding the memory consumption, thanks for putting those numbers > together! I would also love to get #929 > <https://github.com/apache/iceberg-python/pull/929>, so we can push down > the large/small type to PyArrow (only for to_arrow), and apply #986 > <https://github.com/apache/iceberg-python/pull/986> on top if you want to > force it to either small or large types. > > WDYT? > > Kind regards, > Fokko > > > Op vr 2 aug 2024 om 19:46 schreef Sung Yun <sun...@apache.org>: > > > Hi folks, > > > > We identified inefficient memory usage hikes with the current way of > > upcasting pyarrow types to large_<type> on read, when reading tables with > > certain characteristics. A detailed set of example benchmarks of this issue > > is on the google document linked on PR #986: > > https://github.com/apache/iceberg-python/pull/986 > > > > The proposed solution introduces a config to override this behavior to use > > small types instead, and I'd like to add this into the patch release to > > give users better control over their memory usage. > > > > Also, this is just a gentle reminder that this DISCUSS thread is still > > open for any new issues that are identified from 0.7.0 release, that we > > should fix in the patch release. > > > > Thank you, > > Sung > > > > On 2024/07/30 23:57:04 Sung Yun wrote: > > > Hi folks, > > > > > > We are starting to compile the list of issues to fix and port into the > > > 0.7.1 release. > > > > > > The current list of known issues is as follows: > > > > > > Fix pydantic warning on table commit: #972 > > > <https://github.com/apache/iceberg-python/pull/972> (thanks for the > > quick > > > fix ndrluis!) > > > Issue when rewriting an unpartitioned table: #979 > > > <https://github.com/apache/iceberg-python/issues/979> > > > Issue when evolving and writing in the same transaction: #980 > > > <https://github.com/apache/iceberg-python/issues/980> > > > > > > Please feel free to respond to this thread with any issues that should be > > > tracked for the patch release. > > > > > > Thank you! > > > Sung > > > > > >
