Hi all,
We've been evaluating Flight for our use, and we're wondering if the protocol is still open to extensions, as having a few application-defined metadata fields would help our use cases a lot. (Apologies if this is a repost - was having issue with the spam filter.) Specifically, in DoGet, having a metadata binary blob in the server->client messages would help implement resumable requests, especially as we have non-monotonically-indexed data streams. This would also help us reuse server-side state if we do have to resume a stream. In DoPut, we think making this call bidirectional would be useful to support application-level ACKs, again to implement resumable uploads. The server would thus have the option to send back an application-defined binary blob at any point during an upload. This is less important, as you could imagine starting a plain gRPC server-streaming call alongside the Flight DoPut call to do the same. But as you can't bind a gRPC and Flight service on the same port/channel, this is somewhat inconvenient. That leads me to the API-level niggles we have; it would be nice to be able to bind gRPC services alongside a Flight service, and conversely be able to reuse a gRPC channel across gRPC and Flight clients, though breaking the hiding of gRPC isn't desirable. Meanwhile, it would be nice to wrap the gRPC server 'awaitTermination' methods, so that we don't have to busy-wait ourselves (as in Java) or have the option to not busy-wait taken away from us (as in C++). In particular, when investigating Python bindings to C++ [0], the fact that FlightServerBase::Run also calls grpc::Server::Wait for you means that Ctrl-C no longer works in Python. Does what we're trying to accomplish make sense? Are there better ways to achieve resumable uploads/downloads in the current protocol? [0]: https://github.com/apache/arrow/pull/3566 Thanks, David
