Welcome Lewis! The use case you outline makes a lot of sense for Arrow to help out with. We don't yet have an AVRO <> Arrow converter written but it is something that would be great to have. We'd all be happy to help if you're interested in taking this on. The new improvements to the Arrow Java APIs (just merged and will be available in 0.8.0) have made this substantially nicer/easier.
On Thu, Nov 16, 2017 at 10:36 AM, Lewis John McGibbney <lewi...@apache.org> wrote: > Hi Folks, > > We've been working on GORA (Generic Object Representation using Avro) for > some years now. https://gora.apache.org > > The framework provides an in-memory data model and persistence for big > data. Gora supports persisting to column stores, key value stores, document > stores, distributed in-memory key/value stores, in-memory data grids, > in-memory caches, distributed multi-model stores, and hybrid in-memory > architectures. > I am intersted in seeing how arrow can be used within GORA and would > appreciate some input from the community here. > > In GORA we maintain the concept of modeling object structure through use > of a JSON (Avro) schema similar to what we see within the Arrow schema > design as documented at http://arrow.apache.org/docs/metadata.html#schemas. > This is encouraging and although I am still learning about Arrow is seems > that it would be a very nice fit for GORA to consider leveraging as a > common data format/layer. > > A few questions I have as follows > 1) Is Avro supported as an Arrow system? > 2) If so, is there any mechanism(s) to transition/copy data which has been > written using Gora (with Avro as the underlying data format) to Arrow > memory format? I see there is mention of "Projects can share functionality > (eg, Parquet-to-Arrow reader)", is such functionality available for Avro? > > Let's start with the above, I can take it from there. > Thanks in advance for any heads up folks, > Lewis >
