> Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL > implementations via pyo3 / fsspec bindings > <https://github.com/apache/opendal/issues/4511>?
Hi, Raschkowski, good question! It's possible. There is an ongoing project developing fsspec bindings for opendal at https://github.com/fsspec/opendalfs. Once complete, we can directly use opendal through fsspec. This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should be able to use opendalfs as an alternative implementation of the fsspec AbstractFileSystem class. On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote: > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL > implementations via pyo3 / fsspec bindings > <https://github.com/apache/opendal/issues/4511>? > > > > *From:* Joe Stein <crypt...@gmail.com> > *Sent:* Thursday, August 1, 2024 3:37 AM > *To:* dev@iceberg.apache.org <dev@iceberg.apache.org> > *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io > > *CAUTION:* This email originates from an external party (outside of > Palantir). If you believe this message is suspicious in nature, please use > the "Report Message" button built into Outlook. > > Kafka did this with librdkafka and was wildly successful. The underlying > bindings being in rust are great with a layer for access in Python +1 > > > ~ Joe Stein > > > On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xua...@apache.org> wrote: >> Hello everyone >> >> I start this thread to discuss the idea about using iceberg-rust as >> pyiceberg file io. >> >> The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io >> [hackmd.io] >> <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$> >> >> In summary, we can leverage the work from iceberg-rust to help pyiceberg in >> developing a fast and compact file IO system that benefits users with >> specific constraints. >> >> Welcome to join in the discussion. >> >> Xuanwo >> >> https://xuanwo.io/ [xuanwo.io] >> <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$> Xuanwo https://xuanwo.io/
