Thank you for reporting the issues and putting in the fixes Fokko and André.
We also identified a correctness issue with applying positional deletes on merge-on-read tables that I think also must be included into this release. Here's the PR that resolves the issue: https://github.com/apache/iceberg-python/pull/1026 Sung On Thu, Aug 8, 2024 at 9:29 AM André Luis Anastácio <ndrl...@proton.me.invalid> wrote: > I fixed an overwrite error that, I think, would be good to include in the > 0.7.1 release https://github.com/apache/iceberg-python/pull/1023 > > André Anastácio > > On Thursday, August 8th, 2024 at 4:29 AM, Fokko Driesprong < > fo...@apache.org> wrote: > > Thanks everyone for the input here, and I agree that the aforementioned > #995 <https://github.com/apache/iceberg-python/pull/995/> and #997 > <https://github.com/apache/iceberg-python/pull/997/> by Sung, and #526 > <https://github.com/apache/iceberg-python/pull/526> by André would also > be good to include (I've added the milestone there). I have two minor ones > that are also good candidates to add to 0.7.1: > > - Allow setting <http://goog_2004148629>write.parquet.row-group-limit > <https://github.com/apache/iceberg-python/pull/1016> > - Allow setting <http://goog_2004148635>write.parquet.page-row-limit > <https://github.com/apache/iceberg-python/pull/1017> > > Kind regards, > Fokko > > > Op di 6 aug 2024 om 21:17 schreef André Luis Anastácio > <ndrl...@proton.me.invalid>: > >> What do you think about adding the fix that excludes PyIceberg support >> for Python 3.9.7 in the 0.7.1 release?[1] It already doesn't work, so this >> is just to avoid any new issues. >> >> - [1]: https://github.com/apache/iceberg-python/pull/526 >> >> André Anastácio >> >> >> On Tuesday, August 6th, 2024 at 4:06 PM, Sung Yun <sun...@apache.org> >> wrote: >> >> > Sounds good folks! Thank you for sharing your thoughts. We'll work on >> getting the patch release out, and continue the discussion on upgrading the >> PyArrow version to 17.0.0 in time for 0.8.0 release. >> > >> > Just adding these two more fixes that were introduced that I think we >> should pull into the patch release. These were added to the GitHub >> milestone for 0.7.1, but just cross posting here for awareness: >> > >> > - Table scan fails when result is empty: >> https://github.com/apache/iceberg-python/pull/997 >> > - Fix RestCatalog ListNamespace to correctly make use of the expected >> Rest Catalog response: https://github.com/apache/iceberg-python/pull/997 >> > >> > Sung >> > >> > On 2024/08/06 18:29:50 Kevin Liu wrote: >> > >> > > > Typically we only push patches into the minor versions, we could >> also go >> > > > to version 0.8.0 immediately. >> > > >> > > The issues above sound like patches to me, fixing issues discovered >> during >> > > the 0.7.0 release. Is there a reason to move to 0.8.0? >> > > >> > > > I'm still on the fence regarding 17.0.0 upgrade. There are clear >> > > > functional upsides, but I feel that constraining PyIceberg to just >> one >> > > > published version would make the adoption of PyIceberg difficult >> for our >> > > > users. >> > > >> > > +1 on this concern. Is it possible to make the Arrow 17.0.0 upgrade >> > > optional first? So that folks who want the upgrade can test it out. >> > > >> > > Thanks, >> > > Kevin Liu >> > > >> > > On Fri, Aug 2, 2024 at 11:33 AM Sung Yun sun...@apache.org wrote: >> > > >> > > > Hi Fokko, >> > > > >> > > > That makes sense, thank you for the suggestion! The issue was quite >> severe >> > > > for us that we had to fork the repo and have a fix ourselves in >> order to >> > > > run PyIceberg without our applications going OOM. So I think there >> will be >> > > > value in getting the proposed config property out as early as >> possible for >> > > > the larger community. >> > > > >> > > > I'm still on the fence regarding 17.0.0 upgrade. There are clear >> > > > functional upsides, but I feel that constraining PyIceberg to just >> one >> > > > published version would make the adoption of PyIceberg difficult >> for our >> > > > users. Users writing new applications won't have trouble with it, >> but users >> > > > intending to use PyIceberg in an existing application may have to >> upgrade >> > > > their PyArrow versions which could be a deterrent (or a welcome >> nudge). >> > > > Would it be worth starting that discussion on a separate thread? >> > > > >> > > > Sung >> > > > >> > > > On 2024/08/02 17:57:17 Fokko Driesprong wrote: >> > > > >> > > > > Hey Sung, >> > > > > >> > > > > Typically we only push patches into the minor versions, we could >> also go >> > > > > to >> > > > > version 0.8.0 immediately. >> > > > > >> > > > > Regarding the memory consumption, thanks for putting those numbers >> > > > > together! I would also love to get #929 >> > > > > https://github.com/apache/iceberg-python/pull/929, so we can >> push down >> > > > > the large/small type to PyArrow (only for to_arrow), and apply >> #986 >> > > > > https://github.com/apache/iceberg-python/pull/986 on top if you >> want >> > > > > to >> > > > > force it to either small or large types. >> > > > > >> > > > > WDYT? >> > > > > >> > > > > Kind regards, >> > > > > Fokko >> > > > > >> > > > > Op vr 2 aug 2024 om 19:46 schreef Sung Yun sun...@apache.org: >> > > > > >> > > > > > Hi folks, >> > > > > > >> > > > > > We identified inefficient memory usage hikes with the current >> way of >> > > > > > upcasting pyarrow types to large_<type> on read, when reading >> tables >> > > > > > with >> > > > > > certain characteristics. A detailed set of example benchmarks >> of this >> > > > > > issue >> > > > > > is on the google document linked on PR #986: >> > > > > > https://github.com/apache/iceberg-python/pull/986 >> > > > > > >> > > > > > The proposed solution introduces a config to override this >> behavior to >> > > > > > use >> > > > > > small types instead, and I'd like to add this into the patch >> release to >> > > > > > give users better control over their memory usage. >> > > > > > >> > > > > > Also, this is just a gentle reminder that this DISCUSS thread >> is still >> > > > > > open for any new issues that are identified from 0.7.0 release, >> that we >> > > > > > should fix in the patch release. >> > > > > > >> > > > > > Thank you, >> > > > > > Sung >> > > > > > >> > > > > > On 2024/07/30 23:57:04 Sung Yun wrote: >> > > > > > >> > > > > > > Hi folks, >> > > > > > > >> > > > > > > We are starting to compile the list of issues to fix and port >> into >> > > > > > > the >> > > > > > > 0.7.1 release. >> > > > > > > >> > > > > > > The current list of known issues is as follows: >> > > > > > > >> > > > > > > Fix pydantic warning on table commit: #972 >> > > > > > > https://github.com/apache/iceberg-python/pull/972 (thanks >> for the >> > > > > > > quick >> > > > > > > fix ndrluis!) >> > > > > > > Issue when rewriting an unpartitioned table: #979 >> > > > > > > https://github.com/apache/iceberg-python/issues/979 >> > > > > > > Issue when evolving and writing in the same transaction: #980 >> > > > > > > https://github.com/apache/iceberg-python/issues/980 >> > > > > > > >> > > > > > > Please feel free to respond to this thread with any issues >> that >> > > > > > > should be >> > > > > > > tracked for the patch release. >> > > > > > > >> > > > > > > Thank you! >> > > > > > > Sung >> > >
