Hey David, I'm +1 for the protocol changes. I think they make sense. Solving the busy wait in java also makes sense.
On Thu, Feb 14, 2019 at 11:45 AM David Ming Li <david.m...@twosigma.com> wrote: > Back to extending the protocol, all we should need, and the simple thing > (IMO) to do, would be: > > - Add a `bytes data_application = 3` to FlightData ( > https://github.com/apache/arrow/blob/master/format/Flight.proto#L286) > - Add a `bytes data_application = 1` to PutResult > - Change `DoPut` to `rpc DoPut(stream FlightData) returns (stream > PutResult) {}` > > The bigger question is how to change the C++/Java APIs to expose this, as > they kind of assume the only thing around is RecordBatches. > > It does sound interesting to have different underlying transports, which > would then preclude ever exposing gRPC. Would the thought then be to do > token-based authentication in DoGet/DoPut? I suppose the Ticket in DoGet > and the command in DoPut could serve that purpose. > > Best, > David > > On 2019/02/12 20:48:10, Antoine Pitrou <mailto:anto...@python.org> wrote: > > > > Hi David, > > > > I think allowing to send application-specific ancillary data in addition > > to Arrow data makes sense. > > > > (I'm also wondering whether the choice of gRPC is appropriate at all - > > the current C++ hacks around "zero-copy" are not pretty and they may not > > translate to other languages either) > > > > Regards > > > > Antoine. > > > > > > Le 12/02/2019 à 21:44, David Ming Li a écrit : > > > Hi all, > > > > > > > > > > > > We've been evaluating Flight for our use, and we're wondering if the > protocol is still open to extensions, as having a few application-defined > metadata fields would help our use cases a lot. > > > > > > > > > > > > (Apologies if this is a repost - was having issue with the spam > filter.) > > > > > > > > > > > > Specifically, in DoGet, having a metadata binary blob in the > server->client messages would help implement resumable requests, especially > as we have non-monotonically-indexed data streams. This would also help us > reuse server-side state if we do have to resume a stream. > > > > > > > > > > > > In DoPut, we think making this call bidirectional would be useful to > support application-level ACKs, again to implement resumable uploads. The > server would thus have the opt >
