Arrow is mainly about batching data and leveraging all the opportunities this gives. This means you either have to buffer the data yourself and flush it when a reasonable sized batch is complete or play with preallocating Arrow structures This was discussed recently, you might be interested in the thread: https://www.mail-archive.com/dev@arrow.apache.org/msg19862.html
Note: I'm not an Arrow developer, I'm just following the "streaming" features of the Arrow lib, I'm interested in having a "rolling window" API (like a fixed size FIFO queue). Best regards, Adam Lippai On Wed, Aug 12, 2020 at 11:29 AM <m...@markfarnan.com> wrote: > I'm looking at using Arrow for a realtime IoT project which includes use > cases both on server, and also for transferring /using in a Browser via > WASM, and have a few questions. > > > > Language in use is Go. > > > > Is anyone working on implementing Arrow-Flight in Go ? (According to > the feature matrix, nothing ready yet, so wanted to check. > > > > Has anyone tried using Apache Arrow in Go WASM (Webassembly) ? if so, > any issues ? > > > > Any pointers/documentation on using/extending Arrow for realtime streaming > cases. (Specifically where a DataFrame is requested, but then it needs to > 'grow' as new data arrives, often at high speed). > > Not language specific, just trying to understand the right pattern for > using > Arrow for this, and couldn't' find much in the docs. > > > > Regards > > > > Mark. > >
