Hi Micah, The primary language is Python. I'm hoping the that the small overhead of metadata is small compared to the schema information.
thank you! On Fri, Feb 14, 2020 at 3:07 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > Hi Tewfik, > What language? it is possible to serialize them separately but the right > hooks might not be exposed in all languages. > > There is still going to be a higher overhead for single row values in Arrow > compared to Avro due to metadata requirements. > > Thanks, > Micah > > On Fri, Feb 14, 2020 at 1:33 PM Tewfik Zeghmi <zeg...@gmail.com> wrote: > > > Hi, > > > > I have a use case of creating a feature store to serve low latency > traffic. > > Given a key, we need the ability to save and read a feature vector in a > low > > latency Key Value store. Serializing an Arrow table with one row is takes > > 1344 bytes, while the same singular row serialized with AVRO without the > > schema uses 236 bytes. > > > > Is it possible to save serialize an Arrow table/RecordBatch independently > > of the schema? Ideally, we'd like to serialize the schema once and not > > along with every feature key, then be able to read the RecordBatch with > the > > schema. > > > > thank you! > > > -- Taleb Tewfik Zeghmi
