> Decoupling pyiceberg_core with iceberg-rust may be flexible, but my concern > is that this may not be scalable when we introduce more language bindings. I > think Xuanwo has a lot of experience when maintaining Apache OpenDAL > <https://github.com/apache/opendal> ?
OpenDAL tends to release all packages simultaneously but with different versions. We can try this approach as well. It's also fine for me to release pyiceberg_core separately for the first time. On Thu, Aug 29, 2024, at 10:51, Renjie Liu wrote: > Hi: > > Thanks Sung for driving this. > >> Which features do we expose in the first release? > > I think the transformation is a good start. > >> Should the version of the pyiceberg_core python package be aligned with the >> version of iceberg-rust crate? > > Decoupling pyiceberg_core with iceberg-rust may be flexible, but my concern > is that this may not be scalable when we introduce more language bindings. I > think Xuanwo has a lot of experience when maintaining Apache OpenDAL > <https://github.com/apache/opendal> ? > >> What is the release cadence we want to settle on in the future? > > I agree that it's valuable to invest in automation workflow for this. > > > On Wed, Aug 28, 2024 at 10:19 PM Fokko Driesprong <fo...@apache.org> wrote: >> Thanks for driving this Sung, this is very exciting! >> >> 1. The transforms are a good first thing to address. >> 2. I agree with Xuanwo, that for flexibility we can decouple them. >> 3. Automation is probably easier than doing it manually (otherwise we >> would have to document the steps). >> >> Kind regards, >> Fokko >> >> Op di 27 aug 2024 om 17:50 schreef Xuanwo <xua...@apache.org>: >> > >> > Thanks a lot for carrying this! >> > >> > 1. Which features do we expose in the first release? >> > >> > >> > No further comments on this. Pyiceberg is currently the sole user of >> > pyiceberg_core. I'm willing to meet whatever pyiceberg's needs. >> > >> > 2. Should the version of the pyiceberg_core python package be aligned with >> > the version of iceberg-rust crate? >> > >> > >> > I believe pyiceberg_core should have its own version. The release workflow >> > should be very different from the existing Rust workflow, so we don't need >> > to align the versions just for this. >> > >> > 3. We could publish the initial package manually to unblock PyIceberg, but >> > we should invest in a unified release process on github workflows that >> > publishes all enabled language bindings in the repository together. >> > >> > >> > Releasing a Rust-core-based Python package manually is very tedious; it's >> > better to start with a GitHub workflow from the first version. >> > >> > I have experience in building OpenDAL Python workflows, and I'm willing to >> > help establish the release workflow for PyIceberg Core. >> > >> > See: >> > https://github.com/apache/opendal/blob/main/.github/workflows/release_python.yml >> > >> > >> > On Tue, Aug 27, 2024, at 22:56, Sung Yun wrote: >> > >> > Hi folks, >> > >> > We’ve recently had a few discussion threads on components we wanted to >> > expose as Python Bindings from the iceberg-rust project in the form of the >> > "pyiceberg_core" package. >> > >> > Now that the Python bindings for the Transforms have been implemented, I >> > wanted to open this thread to discuss the first 'pyiceberg_core' release. >> > Here are some questions I wanted to discuss with the community: >> > >> > 1. Which features do we expose in the first release? >> > 2. Should the version of the pyiceberg_core python package be aligned with >> > the version of iceberg-rust crate? >> > 3. What is the release cadence we want to settle on in the future? >> > >> > Here are my answers to the above questions, where my thoughts on the >> > versioning and release process are inspired by the current practice from >> > Apache Arrow community <https://arrow.apache.org/release/>: >> > 1. Expose Transforms as Python bindings is ready to release, and we could >> > group in other features if they are also ready: >> > https://github.com/apache/iceberg-rust/pull/556 >> > 2. I think we could start with the current rust version (0.3.0). Aligning >> > it with the version of the iceberg-rust crate would make it easier to set >> > up a CI worklow that releases packages for all available bindings from the >> > same trigger. >> > 3. We could publish the initial package manually to unblock PyIceberg, but >> > we should invest in a unified release process on github workflows that >> > publishes all enabled language bindings in the repository together. >> > >> > I'm excited to hear your thoughts! >> > >> > Sung >> > >> > Prior Discussion Threads: >> > Transforms: >> > https://lists.apache.org/thread/33c0nkc3k6646lvro1lv22pvhwlp50ss >> > FileIO: https://lists.apache.org/thread/86zotqs1wqxojt4zx8np29q5doj1l1wc >> > >> > PR for exposing transforms as Python bindings: >> > https://github.com/apache/iceberg-rust/pull/556 >> > >> > Xuanwo >> > >> > https://xuanwo.io/ >> > Xuanwo https://xuanwo.io/
