I am curious what you accomplish with Arrow + Flight from the browser. Right now, Flight is all gRPC-based, and browser compatibility is a bit mixed. I expect the various transcoders/gRPC-Web can handle GetFlightInfo/DoGet fine, though IIRC for DoGet, at least some of the transcoders would have to buffer the entire stream before sending it to the browser. DoPut/DoExchange seem harder/impossible to bridge right now due to the bidirectional streaming.
Best, David On 8/14/20, m...@markfarnan.com <m...@markfarnan.com> wrote: > Thanks Wes, > > I'll likely work on that once I get my head around Arrow in general and > confirm will use for the project. > > Considerations for how to account for the streaming append problem to an > otherwise immutable dataset is current concern. Still thinking through > that. > > Regards > > Mark. > > -----Original Message----- > From: Wes McKinney <wesmck...@gmail.com> > Sent: Wednesday, August 12, 2020 3:59 PM > To: dev <dev@arrow.apache.org> > Subject: Re: Arrow Flight + Go, Arrow for Realtime > > There's a WIP patch for Flight support in Go > > https://github.com/apache/arrow/pull/6731 > > I hope to see someone taking up this work as first-class Flight support in > Go would be very useful for building data services. > > On Wed, Aug 12, 2020 at 5:08 AM Adam Lippai <a...@rigo.sk> wrote: >> >> Arrow is mainly about batching data and leveraging all the >> opportunities this gives. >> This means you either have to buffer the data yourself and flush it >> when a reasonable sized batch is complete or play with preallocating >> Arrow structures This was discussed recently, you might be interested >> in the thread: >> https://www.mail-archive.com/dev@arrow.apache.org/msg19862.html >> >> Note: I'm not an Arrow developer, I'm just following the "streaming" >> features of the Arrow lib, I'm interested in having a "rolling window" >> API (like a fixed size FIFO queue). >> >> Best regards, >> Adam Lippai >> >> On Wed, Aug 12, 2020 at 11:29 AM <m...@markfarnan.com> wrote: >> >> > I'm looking at using Arrow for a realtime IoT project which includes >> > use cases both on server, and also for transferring /using in a >> > Browser via WASM, and have a few questions. >> > >> > >> > >> > Language in use is Go. >> > >> > >> > >> > Is anyone working on implementing Arrow-Flight in Go ? (According >> > to >> > the feature matrix, nothing ready yet, so wanted to check. >> > >> > >> > >> > Has anyone tried using Apache Arrow in Go WASM (Webassembly) ? if >> > so, >> > any issues ? >> > >> > >> > >> > Any pointers/documentation on using/extending Arrow for realtime >> > streaming >> > cases. (Specifically where a DataFrame is requested, but then it needs >> > to >> > 'grow' as new data arrives, often at high speed). >> > >> > Not language specific, just trying to understand the right pattern >> > for using Arrow for this, and couldn't' find much in the docs. >> > >> > >> > >> > Regards >> > >> > >> > >> > Mark. >> > >> > > >
