I agree that this should be a separate project, so that this can be used by other databases written in rust, not only datafusion. Let's start with an implementation by binding with gandiva, and build pure rust implementation later.
On Sat, May 11, 2019 at 10:28 PM Andy Grove <andygrov...@gmail.com> wrote: > Hi Renjie, > > I have not started on this but I would be interested in helping you with > it. > > At a high level I think there are two main parts to this work: > > 1. Translating DataFusion expressions to Gandiva protobuf > 2. Implementing the code to make the native C call to Gandiva > > I could help with #1 pretty easily. > > I am concerned about the packaging implications of this. I feel quite > strongly that there should be a "pure Rust" version of DataFusion/Arrow and > that the Gandiva integration should be opt-in somehow, so maybe this is a > separate project within the repository, or a feature that can be controlled > by a feature flag in the Cargo.toml somehow. > > Thanks, > > Andy. > > > > > On Sat, May 11, 2019 at 3:10 AM Renjie Liu <liurenjie2...@gmail.com> > wrote: > >> >> Hi: >> @Andy Grove <andygrov...@gmail.com> Are you developing this? I'm >> interested in this and want to join development. >> >> On Tue, Jan 8, 2019 at 3:18 PM Praveen Kumar <prav...@dremio.com> wrote: >> >>> Agree with Wes, the protobuf based interface should be the language >>> neutral >>> way to build expressions with Gandiva. >>> >>> On Mon, Jan 7, 2019 at 8:30 PM Andy Grove <andygrov...@gmail.com> wrote: >>> >>> > This makes sense to me know that I understand a little more about >>> Gandiva. >>> > This also fits well with my proposal to donate DataFusion in the other >>> > thread. DataFusion can manage the overall logical query plan in Rust >>> and >>> > potentially delegate some subset of expression evaluation to Gandiva >>> via >>> > protobuf. >>> > >>> > Thanks, >>> > >>> > Andy. >>> > >>> > On Mon, Jan 7, 2019 at 7:51 AM Wes McKinney <wesmck...@gmail.com> >>> wrote: >>> > >>> > > Gandiva supports a Protobuf-based interface -- this is how Java >>> > > interacts with it via JNI. Rust could do the same -- that would >>> > > probably be easier than wrapping the C++ class structure. It would >>> > > also help drive new feature requirements in the serialized >>> > > projection/filter expression trees >>> > > >>> > > - Wes >>> > > >>> > > On Mon, Jan 7, 2019 at 3:22 AM Krisztián Szűcs >>> > > <szucs.kriszt...@gmail.com> wrote: >>> > > > >>> > > > I'm not sure, that a binding is a good idea. Both Arrow and Parquet >>> > > > already have their own rust implementation, and a interfacing with >>> > > > cpp isn't as easy and straightforward than it is with C. Otherwise >>> > > > We could simply just maintain bindings for all of the cpp >>> libraries, >>> > > > rather than of having a hybrid solution. >>> > > > >>> > > > While We could spare the reimplementation of gandiva, it'd make >>> > > > packaging more complicated and rust development way less >>> > > > welcoming to new contributors. >>> > > > >>> > > > On Fri, Jan 4, 2019 at 3:39 PM Andy Grove <andygrov...@gmail.com> >>> > wrote: >>> > > > >>> > > > > Now that the Rust implementation of Arrow is maturing, I'm >>> interested >>> > > in >>> > > > > having bindings for Gandiva for query execution, rather than >>> > > duplicating >>> > > > > this in Rust. >>> > > > > >>> > > > > I will likely start looking at this soon but wanted to see if >>> anyone >>> > > else >>> > > > > here is particularly interested in this area of functionality? >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > Andy. >>> > > > > >>> > > >>> > >>> >> >> >> -- >> Renjie Liu >> Software Engineer, MVAD >> > -- Renjie Liu Software Engineer, MVAD
