Thanks for starting that, Andy! > I also think it could be helpful with in-memory language interoperability, > such as passing query plans between Python and Rust.
Yes! I prototyped a datafusion-python and pyarrow datasets integration[1] a few weeks ago that could really benefit from this. I'll have to look into it more, but will likely also need a method on PyArrow compute expressions to convert to a Substrait expression. [1] https://github.com/datafusion-contrib/datafusion-python/pull/21 On Mon, Mar 7, 2022 at 8:40 AM Wang Xudong <wxd963996...@gmail.com> wrote: > Thank you! > This is a great idea, I'll try to contribute some code when I have time！ > > --- > xudong > > Gavin Ray <ray.gavi...@gmail.com> 于2022年3月8日周二 00:36写道： > > > Incredibly exciting! Following along eagerly =) > > > > On Mon, Mar 7, 2022 at 11:31 AM Andy Grove <andygrov...@gmail.com> > wrote: > > > > > I created a new repo in the datafusion-contrib GitHub org over the > > weekend > > > with a starting point for supporting DataFusion as both a producer and > > > consumer of Substrait plans. > > > > > > https://github.com/datafusion-contrib/datafusion-substrait > > > > > > I am hopeful that we can eventually use Substrait in Ballista as a > > > replacement for the current query plan protobuf format, meaning that > the > > > Ballista scheduler could potentially be used with engines other than > > > DataFusion. > > > > > > I also think it could be helpful with in-memory language > > interoperability, > > > such as passing query plans between Python and Rust. > > > > > > I plan on continuing to merge my own PRs here as I flesh out more of > > this, > > > at least until there are other contributors. > > > > > > Thanks, > > > > > > Andy. > > > > > >
