Even if zeromq did make more sense, we couldn't take it on as a dependency because of non-ASF-compatible licenses
Java zeromq: MPL 2.0 libzmq: GPL On Tue, Feb 12, 2019 at 3:33 PM Jonathan Chiang <chiang...@gmail.com> wrote: > > Would zeromq make more sense than gRPC? > > Thanks, > Jonathan > > > On Feb 12, 2019, at 12:48 PM, Antoine Pitrou <anto...@python.org> wrote: > > > > > > Hi David, > > > > I think allowing to send application-specific ancillary data in addition > > to Arrow data makes sense. > > > > (I'm also wondering whether the choice of gRPC is appropriate at all - > > the current C++ hacks around "zero-copy" are not pretty and they may not > > translate to other languages either) > > > > Regards > > > > Antoine. > > > > > >> Le 12/02/2019 à 21:44, David Ming Li a écrit : > >> Hi all, > >> > >> > >> > >> We've been evaluating Flight for our use, and we're wondering if the > >> protocol is still open to extensions, as having a few application-defined > >> metadata fields would help our use cases a lot. > >> > >> > >> > >> (Apologies if this is a repost - was having issue with the spam filter.) > >> > >> > >> > >> Specifically, in DoGet, having a metadata binary blob in the > >> server->client messages would help implement resumable requests, > >> especially as we have non-monotonically-indexed data streams. This would > >> also help us reuse server-side state if we do have to resume a stream. > >> > >> > >> > >> In DoPut, we think making this call bidirectional would be useful to > >> support application-level ACKs, again to implement resumable uploads. The > >> server would thus have the option to send back an application-defined > >> binary blob at any point during an upload. This is less important, as you > >> could imagine starting a plain gRPC server-streaming call alongside the > >> Flight DoPut call to do the same. But as you can't bind a gRPC and Flight > >> service on the same port/channel, this is somewhat inconvenient. > >> > >> > >> > >> That leads me to the API-level niggles we have; it would be nice to be > >> able to bind gRPC services alongside a Flight service, and conversely be > >> able to reuse a gRPC channel across gRPC and Flight clients, though > >> breaking the hiding of gRPC isn't desirable. > >> > >> > >> > >> Meanwhile, it would be nice to wrap the gRPC server 'awaitTermination' > >> methods, so that we don't have to busy-wait ourselves (as in Java) or have > >> the option to not busy-wait taken away from us (as in C++). In particular, > >> when investigating Python bindings to C++ [0], the fact that > >> FlightServerBase::Run also calls grpc::Server::Wait for you means that > >> Ctrl-C no longer works in Python. > >> > >> > >> > >> Does what we're trying to accomplish make sense? Are there better ways to > >> achieve resumable uploads/downloads in the current protocol? > >> > >> > >> > >> [0]: https://github.com/apache/arrow/pull/3566 > >> > >> > >> > >> Thanks, > >> > >> David > >> > >>
