Re: [C++] Async Arrow Flight

2021-06-22 Thread David Li
Hey Nate, Thanks for digging this up. This looks reasonable - Arrow already has a Future type. I think we could do something similar, but encapsulate the CompletionQueue inside the Flight client so as to not expose gRPC implementation details. (I'm actually unsure why their SDK exposes this in the

Re: [C++] Async Arrow Flight

2021-06-21 Thread Nate Bauernfeind
Google Cloud supports asynchronous grpc in C++ for parts of their API. One such client-facing API method is this: ``` future> AsyncPublish( google::cloud::CompletionQueue& cq, std::unique_ptr context, google::pubsub::v1::PublishRequest const& request) override { return cq.M

Re: [C++] Async Arrow Flight

2021-06-03 Thread David Li
I see, thanks. That sounds reasonable, and indeed if you are subscribing to lots of data sources (instead of just trying to maximize throughput as I think most people have done before) async may be helpful. There are actually two features being described here though: 1. The C++ Flight client n

Re: [C++] Async Arrow Flight

2021-06-03 Thread Nate Bauernfeind
In addition to Arrow Flight we have other gRPC APIs that work together as a whole. For example, the API client establishes a session with the server. Then the client tells the server to create a derivative data stream by filtering/joining/computing/etc several source streams. The server will keep t

Re: [C++] Async Arrow Flight

2021-06-02 Thread David Li
Hey Nate, I think there's an open JIRA for something like this. I'd love to have something that plays nicely with asyncio/trio in Python and is hopefully more efficient. (I think it would also let us finally have per-message timeouts instead of only a per-call deadline.) There are some challeng

[C++] Async Arrow Flight

2021-06-02 Thread Nate Bauernfeind
It seems to me that the c++ arrow flight implementation uses only the synchronous version of the gRPC API. gRPC supports asynchronous message delivery in C++ via a CompletionQueue that must be polled. Has there been any desire to standardize on a solution for asynchronous use cases, perhaps deliver