Thanks Wes, I'll likely work on that once I get my head around Arrow in general and confirm will use for the project.
Considerations for how to account for the streaming append problem to an otherwise immutable dataset is current concern. Still thinking through that. Regards Mark. -----Original Message----- From: Wes McKinney <wesmck...@gmail.com> Sent: Wednesday, August 12, 2020 3:59 PM To: dev <dev@arrow.apache.org> Subject: Re: Arrow Flight + Go, Arrow for Realtime There's a WIP patch for Flight support in Go https://github.com/apache/arrow/pull/6731 I hope to see someone taking up this work as first-class Flight support in Go would be very useful for building data services. On Wed, Aug 12, 2020 at 5:08 AM Adam Lippai <a...@rigo.sk> wrote: > > Arrow is mainly about batching data and leveraging all the > opportunities this gives. > This means you either have to buffer the data yourself and flush it > when a reasonable sized batch is complete or play with preallocating > Arrow structures This was discussed recently, you might be interested > in the thread: > https://www.mail-archive.com/dev@arrow.apache.org/msg19862.html > > Note: I'm not an Arrow developer, I'm just following the "streaming" > features of the Arrow lib, I'm interested in having a "rolling window" > API (like a fixed size FIFO queue). > > Best regards, > Adam Lippai > > On Wed, Aug 12, 2020 at 11:29 AM <m...@markfarnan.com> wrote: > > > I'm looking at using Arrow for a realtime IoT project which includes > > use cases both on server, and also for transferring /using in a > > Browser via WASM, and have a few questions. > > > > > > > > Language in use is Go. > > > > > > > > Is anyone working on implementing Arrow-Flight in Go ? (According to > > the feature matrix, nothing ready yet, so wanted to check. > > > > > > > > Has anyone tried using Apache Arrow in Go WASM (Webassembly) ? if so, > > any issues ? > > > > > > > > Any pointers/documentation on using/extending Arrow for realtime streaming > > cases. (Specifically where a DataFrame is requested, but then it needs to > > 'grow' as new data arrives, often at high speed). > > > > Not language specific, just trying to understand the right pattern > > for using Arrow for this, and couldn't' find much in the docs. > > > > > > > > Regards > > > > > > > > Mark. > > > >
