We do have relational operators as well in our code. We're trying to figure out what to contribute back and how to factor. For now, the code is under Apache license you are free to use. Our relational operations are under here:
https://github.com/dremio/dremio-oss/tree/master/sabot/kernel/src/main/java/com/dremio/sabot/op For example, you can see how we do a columnar pivot for the purposes of aggregation here: https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/sabot/op/aggregate/vectorized/VectorizedHashAggOperator.java and here (pivotVariableLengths is especially fun): https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/sabot/op/common/ht2/Pivots.java Our goal is ultimately to make Sabot componentized enough that you can use pieces as a library but it will take some time to get all the way there. On Fri, Jul 21, 2017 at 4:35 AM, Michael Shtelma <mshte...@gmail.com> wrote: > Hi Wes, > > It is really great, that you have open-sourced all this! > As far as I understand, you have also open-sourced the engine that can > execute relational operators on arrow ? > Is it possible to use it as library ? > Are you also planning to donate it arrow project at some point? > > Sincerely, > Michael Shtelma > > On Thu, Jul 20, 2017 at 10:19 PM, Wes McKinney <wesmck...@gmail.com> > wrote: > > > hi Sven, > > > > There is a placeholder project in apache/parquet-mr > > https://github.com/apache/parquet-mr/tree/master/parquet-arrow. > > > > It appears in the meantime that Dremio has created a vectorized > > Parquet <-> Arrow reader/writer which has just been open sourced under > > ASL 2.0: https://github.com/dremio/dremio-oss/tree/master/sabot/ > > kernel/src/main/java/com/dremio/exec/store/parquet > > > > I am sure they are very busy right now, but it may be worth discussing > > factoring out this Parquet <-> Arrow interface into a library > > component that can be donated to Apache Parquet. > > > > - Wes > > > > On Wed, Jul 19, 2017 at 4:28 PM, Sven Wagner-Boysen > > <sven.wagner-boy...@signavio.com> wrote: > > > Hi, > > > > > > I started looking into the projects Parquet and Arrow. Looks very > > promising > > > to me. > > > > > > I also came across PyArrow and the Parquet-Arrow integration in Python. > > Is > > > there something similar available for Java? > > > > > > https://arrow.apache.org/docs/python/parquet.html > > > > > > Thanks > > > Sven > > >
