David, Still investigating, but I suspect for streaming I may have to fall back to some form of "custom" Flight implementation over Websockets.
Assuming Arrow/Flight actually makes sense for that link, which will probably depend on how well it compresses. However it will be very nice if it does, to allow common format everywhere. The data I need to move around is highly variable in 'type', (Arrays of Floats, Ints & Strings with occasional Binary, or vector (array of an array of floats in my case) but the number of columns, and their type vary by dataset and visualization choices. So far arrow seems a good choice rather than any 'roll your own', and it will be nice to use same format on Client side as well as in the Server system. My use case is primarily 'Get', consuming large datasets for visualization. I doubt I'll need Put or Exchange from the browser. Mark. -----Original Message----- From: David Li <li.david...@gmail.com> Sent: Saturday, August 15, 2020 5:53 PM To: dev@arrow.apache.org Subject: Re: Arrow Flight + Go, Arrow for Realtime I am curious what you accomplish with Arrow + Flight from the browser. Right now, Flight is all gRPC-based, and browser compatibility is a bit mixed. I expect the various transcoders/gRPC-Web can handle GetFlightInfo/DoGet fine, though IIRC for DoGet, at least some of the transcoders would have to buffer the entire stream before sending it to the browser. DoPut/DoExchange seem harder/impossible to bridge right now due to the bidirectional streaming. Best, David On 8/14/20, m...@markfarnan.com <m...@markfarnan.com> wrote: > Thanks Wes, > > I'll likely work on that once I get my head around Arrow in general > and confirm will use for the project. > > Considerations for how to account for the streaming append problem to an > otherwise immutable dataset is current concern. Still thinking through > that. > > Regards > > Mark. > > -----Original Message----- > From: Wes McKinney <wesmck...@gmail.com> > Sent: Wednesday, August 12, 2020 3:59 PM > To: dev <dev@arrow.apache.org> > Subject: Re: Arrow Flight + Go, Arrow for Realtime > > There's a WIP patch for Flight support in Go > > https://github.com/apache/arrow/pull/6731 > > I hope to see someone taking up this work as first-class Flight > support in Go would be very useful for building data services. > > On Wed, Aug 12, 2020 at 5:08 AM Adam Lippai <a...@rigo.sk> wrote: >> >> Arrow is mainly about batching data and leveraging all the >> opportunities this gives. >> This means you either have to buffer the data yourself and flush it >> when a reasonable sized batch is complete or play with preallocating >> Arrow structures This was discussed recently, you might be interested >> in the thread: >> https://www.mail-archive.com/dev@arrow.apache.org/msg19862.html >> >> Note: I'm not an Arrow developer, I'm just following the "streaming" >> features of the Arrow lib, I'm interested in having a "rolling window" >> API (like a fixed size FIFO queue). >> >> Best regards, >> Adam Lippai >> >> On Wed, Aug 12, 2020 at 11:29 AM <m...@markfarnan.com> wrote: >> >> > I'm looking at using Arrow for a realtime IoT project which >> > includes use cases both on server, and also for transferring /using >> > in a Browser via WASM, and have a few questions. >> > >> > >> > >> > Language in use is Go. >> > >> > >> > >> > Is anyone working on implementing Arrow-Flight in Go ? (According >> > to >> > the feature matrix, nothing ready yet, so wanted to check. >> > >> > >> > >> > Has anyone tried using Apache Arrow in Go WASM (Webassembly) ? if >> > so, >> > any issues ? >> > >> > >> > >> > Any pointers/documentation on using/extending Arrow for realtime >> > streaming >> > cases. (Specifically where a DataFrame is requested, but then it needs >> > to >> > 'grow' as new data arrives, often at high speed). >> > >> > Not language specific, just trying to understand the right pattern >> > for using Arrow for this, and couldn't' find much in the docs. >> > >> > >> > >> > Regards >> > >> > >> > >> > Mark. >> > >> > > >
