Re: Migrating from Avro to Arrow

2017-11-17 Thread Lewis John McGibbney
Ideal, I'll let you know how we get on. Thank you On 2017-11-16 14:45, Jacques Nadeau wrote: > For java, you can start by looking at this entry point: > > https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/store/parquet/columnreaders/DeprecatedParquetVec

Re: Migrating from Avro to Arrow

2017-11-16 Thread Jacques Nadeau
For java, you can start by looking at this entry point: https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/store/parquet/columnreaders/DeprecatedParquetVectorizedReader.java Something that might actually be easier as an initial understanding (simpler) is l

Re: Migrating from Avro to Arrow

2017-11-16 Thread Lewis John McGibbney
Hi Jacques, Can you point me to where I get started e.g. with the converter? Where does the Parquet --> Arrow one current exist? Thank you On 2017-11-16 10:42, Jacques Nadeau wrote: > Welcome Lewis! > > The use case you outline makes a lot of sense for Arrow to help out > with. We don't yet hav

Re: Migrating from Avro to Arrow

2017-11-16 Thread Jacques Nadeau
Welcome Lewis! The use case you outline makes a lot of sense for Arrow to help out with. We don't yet have an AVRO <> Arrow converter written but it is something that would be great to have. We'd all be happy to help if you're interested in taking this on. The new improvements to the Arrow Java AP

Migrating from Avro to Arrow

2017-11-16 Thread Lewis John McGibbney
Hi Folks, We've been working on GORA (Generic Object Representation using Avro) for some years now. https://gora.apache.org The framework provides an in-memory data model and persistence for big data. Gora supports persisting to column stores, key value stores, document stores, distributed in-