Hi: Thanks Sung for driving this.
Which features do we expose in the first release? I think the transformation is a good start. Should the version of the pyiceberg_core python package be aligned with the > version of iceberg-rust crate? Decoupling pyiceberg_core with iceberg-rust may be flexible, but my concern is that this may not be scalable when we introduce more language bindings. I think Xuanwo has a lot of experience when maintaining Apache OpenDAL <https://github.com/apache/opendal> ? What is the release cadence we want to settle on in the future? I agree that it's valuable to invest in automation workflow for this. On Wed, Aug 28, 2024 at 10:19 PM Fokko Driesprong <fo...@apache.org> wrote: > Thanks for driving this Sung, this is very exciting! > > 1. The transforms are a good first thing to address. > 2. I agree with Xuanwo, that for flexibility we can decouple them. > 3. Automation is probably easier than doing it manually (otherwise we > would have to document the steps). > > Kind regards, > Fokko > > Op di 27 aug 2024 om 17:50 schreef Xuanwo <xua...@apache.org>: > > > > Thanks a lot for carrying this! > > > > 1. Which features do we expose in the first release? > > > > > > No further comments on this. Pyiceberg is currently the sole user of > pyiceberg_core. I'm willing to meet whatever pyiceberg's needs. > > > > 2. Should the version of the pyiceberg_core python package be aligned > with the version of iceberg-rust crate? > > > > > > I believe pyiceberg_core should have its own version. The release > workflow should be very different from the existing Rust workflow, so we > don't need to align the versions just for this. > > > > 3. We could publish the initial package manually to unblock PyIceberg, > but we should invest in a unified release process on github workflows that > publishes all enabled language bindings in the repository together. > > > > > > Releasing a Rust-core-based Python package manually is very tedious; > it's better to start with a GitHub workflow from the first version. > > > > I have experience in building OpenDAL Python workflows, and I'm willing > to help establish the release workflow for PyIceberg Core. > > > > See: > https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml > > > > > > On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote: > > > > Hi folks, > > > > We’ve recently had a few discussion threads on components we wanted to > expose as Python Bindings from the iceberg-rust project in the form of the > "pyiceberg_core" package. > > > > Now that the Python bindings for the Transforms have been implemented, I > wanted to open this thread to discuss the first 'pyiceberg_core' release. > Here are some questions I wanted to discuss with the community: > > > > 1. Which features do we expose in the first release? > > 2. Should the version of the pyiceberg_core python package be aligned > with the version of iceberg-rust crate? > > 3. What is the release cadence we want to settle on in the future? > > > > Here are my answers to the above questions, where my thoughts on the > versioning and release process are inspired by the current practice from > Apache Arrow community <https://arrow.apache.org/release/>: > > 1. Expose Transforms as Python bindings is ready to release, and we > could group in other features if they are also ready: > https://github.com/apache/iceberg-rust/pull/556 > > 2. I think we could start with the current rust version (0.3.0). > Aligning it with the version of the iceberg-rust crate would make it easier > to set up a CI worklow that releases packages for all available bindings > from the same trigger. > > 3. We could publish the initial package manually to unblock PyIceberg, > but we should invest in a unified release process on github workflows that > publishes all enabled language bindings in the repository together. > > > > I'm excited to hear your thoughts! > > > > Sung > > > > Prior Discussion Threads: > > Transforms: > https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss > > FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc > > > > PR for exposing transforms as Python bindings: > https://github.com/apache/iceberg-rust/pull/556 > > > > Xuanwo > > > > https://xuanwo.io/ > > >
