Back to extending the protocol, all we should need, and the simple thing (IMO) to do, would be:
- Add a `bytes data_application = 3` to FlightData (https://github.com/apache/arrow/blob/master/format/Flight.proto#L286) - Add a `bytes data_application = 1` to PutResult - Change `DoPut` to `rpc DoPut(stream FlightData) returns (stream PutResult) {}` The bigger question is how to change the C++/Java APIs to expose this, as they kind of assume the only thing around is RecordBatches. It does sound interesting to have different underlying transports, which would then preclude ever exposing gRPC. Would the thought then be to do token-based authentication in DoGet/DoPut? I suppose the Ticket in DoGet and the command in DoPut could serve that purpose. Best, David On 2019/02/12 20:48:10, Antoine Pitrou <mailto:anto...@python.org> wrote: > > Hi David, > > I think allowing to send application-specific ancillary data in addition > to Arrow data makes sense. > > (I'm also wondering whether the choice of gRPC is appropriate at all - > the current C++ hacks around "zero-copy" are not pretty and they may not > translate to other languages either) > > Regards > > Antoine. > > > Le 12/02/2019 à 21:44, David Ming Li a écrit : > > Hi all, > > > > > > > > We've been evaluating Flight for our use, and we're wondering if the > > protocol is still open to extensions, as having a few application-defined > > metadata fields would help our use cases a lot. > > > > > > > > (Apologies if this is a repost - was having issue with the spam filter.) > > > > > > > > Specifically, in DoGet, having a metadata binary blob in the server->client > > messages would help implement resumable requests, especially as we have > > non-monotonically-indexed data streams. This would also help us reuse > > server-side state if we do have to resume a stream. > > > > > > > > In DoPut, we think making this call bidirectional would be useful to > > support application-level ACKs, again to implement resumable uploads. The > > server would thus have the opt
