Hey Sung, Typically we only push patches into the minor versions, we could also go to version 0.8.0 immediately.
Regarding the memory consumption, thanks for putting those numbers together! I would also love to get #929 <https://github.com/apache/iceberg-python/pull/929>, so we can push down the large/small type to PyArrow (only for to_arrow), and apply #986 <https://github.com/apache/iceberg-python/pull/986> on top if you want to force it to either small or large types. WDYT? Kind regards, Fokko Op vr 2 aug 2024 om 19:46 schreef Sung Yun <sun...@apache.org>: > Hi folks, > > We identified inefficient memory usage hikes with the current way of > upcasting pyarrow types to large_<type> on read, when reading tables with > certain characteristics. A detailed set of example benchmarks of this issue > is on the google document linked on PR #986: > https://github.com/apache/iceberg-python/pull/986 > > The proposed solution introduces a config to override this behavior to use > small types instead, and I'd like to add this into the patch release to > give users better control over their memory usage. > > Also, this is just a gentle reminder that this DISCUSS thread is still > open for any new issues that are identified from 0.7.0 release, that we > should fix in the patch release. > > Thank you, > Sung > > On 2024/07/30 23:57:04 Sung Yun wrote: > > Hi folks, > > > > We are starting to compile the list of issues to fix and port into the > > 0.7.1 release. > > > > The current list of known issues is as follows: > > > > Fix pydantic warning on table commit: #972 > > <https://github.com/apache/iceberg-python/pull/972> (thanks for the > quick > > fix ndrluis!) > > Issue when rewriting an unpartitioned table: #979 > > <https://github.com/apache/iceberg-python/issues/979> > > Issue when evolving and writing in the same transaction: #980 > > <https://github.com/apache/iceberg-python/issues/980> > > > > Please feel free to respond to this thread with any issues that should be > > tracked for the patch release. > > > > Thank you! > > Sung > > >
