Yes, you can use a tool like grpcurl to call RPCs on an Arrow Flight server by doing something like this,
grpcurl -plaintext \ -import-path /path/to/clone/of/arrow/format/ \ -proto Flight.proto \ localhost:8815 \ arrow.flight.protocol.FlightService/DoGet On Tue, Mar 18, 2025 at 7:26 AM Z A <z11...@outlook.com> wrote: > > I am thinking to use it as a grpc proxy, so not HTTP <-> Arrow Flight grpc, > but I am still looking into it. Thanks for explaining those patterns. > Can i use grpcurl to call Arrow Flight? > > ________________________________ > From: kekronbekron <kekronbek...@protonmail.com> > Sent: Sunday, March 16, 2025 11:06 PM > To: user@arrow.apache.org <user@arrow.apache.org> > Subject: Re: api gateway with arrow flight grpc > > Yes, an API gateway in front surely helps, especially if you "front" this > with an HTTP server rather than gRPC. > 3 different patterns - > > 1: > API gateway / reverse proxy --> HTTP server --> Flight Server > GET bla.com/pipe --> connects to a duckdb file via Flight and sends stuff over > > 2: > API gateway / reverse proxy --> HTTP server --> Flight SQL > GET bla.com/pipe --> connects to a duckdb file and sends SQL over > Advantage is that you can now connect to the Flight SQL server with JDBC/ADBC > even outside of this app, essentially adding concurrency to the backing > (Duck)DBs... on top of concurrent access to the DB(s) _in_ this program > > 3: > API gateway / reverse proxy --> HTTP server --> ADBC > https://arrow.apache.org/blog/2025/03/10/fast-streaming-inserts-in-duckdb-with-adbc/ > This removes the need for Flight altogether and appends directly into DBs > with the ADBC driver manager. > With HTTP in the front, you can route requests to whichever DB you want. > However, managing replication (or dual write) b/w clones/copies of DBs needs > some thinking. > In DuckDB's case, it's challenging because remote attach over https/s3 is > readonly. > So even if you stream out IPC in your HTTP server's function (for the /pipe > route), you'd need a second HTTP server to receive IPC and write it in... > which is fine and still works. > But do note that this means accessing the DBs entirely via HTTP, so most of > the securty etc will be at the API gateway / HTTP layer. > > > > On Monday, March 17th, 2025 at 06:41, Z A <z11...@outlook.com> wrote: > > Thanks all for the reply and insights! I am going to investigate the grpc > passthru feature which is supported by the API gateway product. > Btw, I wrote a simple Arrow Flight Server in python, but I couldn't figure > out how to call it from grpcurl (all my attempts failed), is that even > possible? > ________________________________ > From: Hélder Gregório <helder.grego...@dremio.com> > Sent: Saturday, March 15, 2025 8:28 AM > To: user@arrow.apache.org <user@arrow.apache.org> > Subject: Re: api gateway with arrow flight grpc > > Hi, > > I'm not sure about integration with the services you mentioned but assuming > some responsibilities of the API Gateway are handling authentication and load > balancing, you can implement a flight proxy by creating a service that > implements FlightProducer interface. > The service can access headers for validation if needed and redirect vectors > it receives to the designated flight server using another flight client. > > Hope it helps > > On Sat, Mar 15, 2025 at 3:03 AM kekronbekron <kekronbek...@protonmail.com> > wrote: > > Sure - > > https://www.definite.app/blog/duck-takes-flight > https://blobs.duckdb.org/events/duckcon6/rusty-conover-airport-for-duckdb-letting-duckdb-take-apache-arrow-flights.pdf > > > > > On Saturday, March 15th, 2025 at 01:25, Bryce Mecum <bryceme...@gmail.com> > wrote: > > > Hi kekronbekron, can you share any pointers to the pattern you mention > > and where people are talking about it? It sounds like something I > > might be interested in tracking. > > > > On Thu, Mar 13, 2025 at 7:27 PM kekronbekron > > kekronbek...@protonmail.com wrote: > > > > > I'm embarking on exactly this. > > > Amusing how this pattern has become "vocal" in the past few weeks, though > > > Flight has existed for a few years now. > > > > > > On Friday, March 14th, 2025 at 03:56, Paul Whalen pgwha...@gmail.com > > > wrote: > > > > > > I haven’t used anything you’ve mentioned, but I have deployed an arrow > > > flight server in an istio/envoy service mesh, and seen all the benefits > > > of gRPC in that context without any trouble. > > > > > > I’ve often though it would be fun to write an envoy filter for the Flight > > > or Flight SQL protocol, but it isn’t clear to me that there’s much demand > > > for it (for comparison, there’s one for Postgres: > > > https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/postgres). > > > > > > Paul > > > > > > On Mar 13, 2025, at 5:05 PM, Z A z11...@outlook.com wrote: > > > > > > Thanks Felipe! > > > ________________________________ > > > From: Felipe Oliveira Carvalho felipe...@gmail.com > > > Sent: Thursday, March 13, 2025 12:29 AM > > > To: user@arrow.apache.org user@arrow.apache.org > > > Subject: Re: api gateway with arrow flight grpc > > > > > > No, but if these are gRPC proxies they should work. > > > > > > On Wed, 12 Mar 2025 at 18:13 Z A z11...@outlook.com wrote: > > > > > > Hi, > > > I just subscribed to this mailing list, and apologize if this is a silly > > > question. > > > Has anyone ever done any integration of API Gateway (i.e. Kong, Tyk, > > > KrakenD, etc.) with your own Arrow Flight Server? > > > Thanks! > >
