Hey Wes et al, Our goal at Dremio was to contribute the project to the Apache Arrow project if people think that makes sense. (So much so that you have noticed many things are already namespaced Apache.) Gandiva is another way we can continue to have Apache projects drive the vision of a deconstructed database.
The additional reality is we are trying to get a bunch of things done for Dremio so we need to figure out how to make sure the current Gandiva developers like Ravindra can stay productive through a transition (they are not Arrow committers). I've got several things going on and traveling a lot over the next little bit but our goal is definitely to share this with community. Look forward to more feedback of whether others also think Arrow would be a good home for this project. (As opposed to other options shall as GitHub managed or a new Apache project.) thanks for all the support! On Thu, Jun 21, 2018 at 4:26 PM, Wes McKinney <wesmck...@gmail.com> wrote: > hi Jacques, > > This is very exciting! LLVM codegen for Arrow has been on my wishlist > since the early days of the project. I always considered it more of a > "when" question more than "if". > > I will take a closer look at the codebase to make some comments, but > my biggest initial question is whether we could work to make Gandiva > the official community-supported LLVM framework for creating > JIT-compiled Arrow kernels. In the Ursa Labs (a new lab I am building > to focus 90+% on Apache Arrow development) tech roadmap we discussed > the need for a subgraph compiler using LLVM: > https://ursalabs.org/tech/#subgraph-compilation-code-generation. > > I would be interesting in getting involved in the project, and I > expect in time many others will, as well. An obvious question would be > whether you would be interested in donating the project to Apache > Arrow and continuing the work there. We would benefit from common > build, testing/CI, and packaging/deployment infrastructure. I'm keen > to see JIT-powered predicate pushdown in Parquet files, for example. > Phillip and I could look into building a Gandiva backend for compiling > a subset of expressions originating from Ibis, a lazy-evaluation DSL > system with similar API to pandas > (https://github.com/ibis-project/ibis). > > best > Wes > > On Thu, Jun 21, 2018 at 4:13 PM, Dimitri Vorona > <alen...@googlemail.com.invalid> wrote: > > Hey Jaques, > > > > Great stuff! I'm actually researching the integration of arrow and flight > > into a main memory database which also uses LLVM for dynamic query > > generation! Excited to have a more detailed look at Gandiva! > > > > Cheers, > > Dimitri. > > > > On Thu, Jun 21, 2018, 21:15 Jacques Nadeau <jacq...@apache.org> wrote: > > > >> Hey Guys, > >> > >> Dremio just open sourced a new framework for processing data in Arrow > data > >> structures [1], built on top of the Apache Arrow C++ APIs and leveraging > >> LLVM (Apache licensed). It also includes Java APIs that leverage the > Apache > >> Arrow Java libraries. I expect the developers who have been working on > this > >> will introduce themselves soon. To read more about it, take a look at > our > >> Ravindra's blog post (he's the lead developer driving this work): [2]. > >> Hopefully people will find this interesting/useful. > >> > >> Let us know what you all think! > >> > >> thanks, > >> Jacques > >> > >> > >> [1] https://github.com/dremio/gandiva > >> [2] https://www.dremio.com/announcing-gandiva-initiative- > for-apache-arrow/ > >> >
