Would zeromq make more sense than gRPC? Thanks, Jonathan
> On Feb 12, 2019, at 12:48 PM, Antoine Pitrou <anto...@python.org> wrote: > > > Hi David, > > I think allowing to send application-specific ancillary data in addition > to Arrow data makes sense. > > (I'm also wondering whether the choice of gRPC is appropriate at all - > the current C++ hacks around "zero-copy" are not pretty and they may not > translate to other languages either) > > Regards > > Antoine. > > >> Le 12/02/2019 à 21:44, David Ming Li a écrit : >> Hi all, >> >> >> >> We've been evaluating Flight for our use, and we're wondering if the >> protocol is still open to extensions, as having a few application-defined >> metadata fields would help our use cases a lot. >> >> >> >> (Apologies if this is a repost - was having issue with the spam filter.) >> >> >> >> Specifically, in DoGet, having a metadata binary blob in the server->client >> messages would help implement resumable requests, especially as we have >> non-monotonically-indexed data streams. This would also help us reuse >> server-side state if we do have to resume a stream. >> >> >> >> In DoPut, we think making this call bidirectional would be useful to support >> application-level ACKs, again to implement resumable uploads. The server >> would thus have the option to send back an application-defined binary blob >> at any point during an upload. This is less important, as you could imagine >> starting a plain gRPC server-streaming call alongside the Flight DoPut call >> to do the same. But as you can't bind a gRPC and Flight service on the same >> port/channel, this is somewhat inconvenient. >> >> >> >> That leads me to the API-level niggles we have; it would be nice to be able >> to bind gRPC services alongside a Flight service, and conversely be able to >> reuse a gRPC channel across gRPC and Flight clients, though breaking the >> hiding of gRPC isn't desirable. >> >> >> >> Meanwhile, it would be nice to wrap the gRPC server 'awaitTermination' >> methods, so that we don't have to busy-wait ourselves (as in Java) or have >> the option to not busy-wait taken away from us (as in C++). In particular, >> when investigating Python bindings to C++ [0], the fact that >> FlightServerBase::Run also calls grpc::Server::Wait for you means that >> Ctrl-C no longer works in Python. >> >> >> >> Does what we're trying to accomplish make sense? Are there better ways to >> achieve resumable uploads/downloads in the current protocol? >> >> >> >> [0]: https://github.com/apache/arrow/pull/3566 >> >> >> >> Thanks, >> >> David >> >>
