I am also very excited for this, especially the possibility of
leveraging it in Ballista. Great work Andy!
On Mon, Mar 7, 2022 at 8:31 AM Andy Grove wrote:
>
> I created a new repo in the datafusion-contrib GitHub org over the weekend
> with a starting point for supporting DataFusion as both a pr
Actually I think I described it backwards. This would be to convert a data
fusion push down filter into an Arrow dataset expression, using substrait
as the intermediate representation.
On Mon, Mar 7, 2022 at 11:52 Weston Pace wrote:
> > but will likely also need a method on PyArrow compute expre
> but will likely also need a method on PyArrow compute expressions to convert
> to a Substrait expression.
There is a C++ method to do this (one of the arrow::engine::ToProto
overloads takes in arrow::compute::Expression and returns
substrait::Expression) but at the moment the method is internal
Thanks for starting that, Andy!
> I also think it could be helpful with in-memory language interoperability,
> such as passing query plans between Python and Rust.
Yes! I prototyped a datafusion-python and pyarrow datasets integration[1] a
few weeks ago that could really benefit from this. I'll h
Thank you!
This is a great idea, I'll try to contribute some code when I have time！
---
xudong
Gavin Ray 于2022年3月8日周二 00:36写道：
> Incredibly exciting! Following along eagerly =)
>
> On Mon, Mar 7, 2022 at 11:31 AM Andy Grove wrote:
>
> > I created a new repo in the datafusion-contrib GitHub org
Incredibly exciting! Following along eagerly =)
On Mon, Mar 7, 2022 at 11:31 AM Andy Grove wrote:
> I created a new repo in the datafusion-contrib GitHub org over the weekend
> with a starting point for supporting DataFusion as both a producer and
> consumer of Substrait plans.
>
> https://githu
I created a new repo in the datafusion-contrib GitHub org over the weekend
with a starting point for supporting DataFusion as both a producer and
consumer of Substrait plans.
https://github.com/datafusion-contrib/datafusion-substrait
I am hopeful that we can eventually use Substrait in Ballista a
