Hi Tewfik, What language? it is possible to serialize them separately but the right hooks might not be exposed in all languages.
There is still going to be a higher overhead for single row values in Arrow compared to Avro due to metadata requirements. Thanks, Micah On Fri, Feb 14, 2020 at 1:33 PM Tewfik Zeghmi <zeg...@gmail.com> wrote: > Hi, > > I have a use case of creating a feature store to serve low latency traffic. > Given a key, we need the ability to save and read a feature vector in a low > latency Key Value store. Serializing an Arrow table with one row is takes > 1344 bytes, while the same singular row serialized with AVRO without the > schema uses 236 bytes. > > Is it possible to save serialize an Arrow table/RecordBatch independently > of the schema? Ideally, we'd like to serialize the schema once and not > along with every feature key, then be able to read the RecordBatch with the > schema. > > thank you! >
