Thanks Bryce! Almost there 🙂 so i can make request to ListFlights, but for
DoGet i need to pass the ticket data
After few attempts, I still can't figure out how to pass the ticket data to
DoGet, I see the Ticket has this proto schema.
message Ticket {
bytes ticket = 1;
}
________________________________
From: Bryce Mecum <bryceme...@gmail.com>
Sent: Thursday, March 20, 2025 8:43 PM
To: user@arrow.apache.org <user@arrow.apache.org>
Subject: Re: api gateway with arrow flight grpc
I had a checkout of the arrow monorepo [1] and the file is at
./format/Flight.proto [2].
[1] https://github.com/apache/arrow/
[2] https://github.com/apache/arrow/blob/main/format/Flight.proto
On Thu, Mar 20, 2025 at 5:40 PM Z A <z11...@outlook.com> wrote:
>
> Hi Bryce,
> Thanks for the reply. Where can i get the path for the "-import-path" and
> also where can i get the file Flight.proto?
> ________________________________
> From: Bryce Mecum <bryceme...@gmail.com>
> Sent: Tuesday, March 18, 2025 12:15 PM
> To: user@arrow.apache.org <user@arrow.apache.org>
> Subject: Re: api gateway with arrow flight grpc
>
> Yes, you can use a tool like grpcurl to call RPCs on an Arrow Flight
> server by doing something like this,
>
> grpcurl -plaintext \
> -import-path /path/to/clone/of/arrow/format/ \
> -proto Flight.proto \
> localhost:8815 \
> arrow.flight.protocol.FlightService/DoGet
>
> On Tue, Mar 18, 2025 at 7:26 AM Z A <z11...@outlook.com> wrote:
> >
> > I am thinking to use it as a grpc proxy, so not HTTP <-> Arrow Flight grpc,
> > but I am still looking into it. Thanks for explaining those patterns.
> > Can i use grpcurl to call Arrow Flight?
> >
> > ________________________________
> > From: kekronbekron <kekronbek...@protonmail.com>
> > Sent: Sunday, March 16, 2025 11:06 PM
> > To: user@arrow.apache.org <user@arrow.apache.org>
> > Subject: Re: api gateway with arrow flight grpc
> >
> > Yes, an API gateway in front surely helps, especially if you "front" this
> > with an HTTP server rather than gRPC.
> > 3 different patterns -
> >
> > 1:
> > API gateway / reverse proxy --> HTTP server --> Flight Server
> > GET bla.com/pipe --> connects to a duckdb file via Flight and sends stuff
> > over
> >
> > 2:
> > API gateway / reverse proxy --> HTTP server --> Flight SQL
> > GET bla.com/pipe --> connects to a duckdb file and sends SQL over
> > Advantage is that you can now connect to the Flight SQL server with
> > JDBC/ADBC even outside of this app, essentially adding concurrency to the
> > backing (Duck)DBs... on top of concurrent access to the DB(s) _in_ this
> > program
> >
> > 3:
> > API gateway / reverse proxy --> HTTP server --> ADBC
> > https://arrow.apache.org/blog/2025/03/10/fast-streaming-inserts-in-duckdb-with-adbc/
> > This removes the need for Flight altogether and appends directly into DBs
> > with the ADBC driver manager.
> > With HTTP in the front, you can route requests to whichever DB you want.
> > However, managing replication (or dual write) b/w clones/copies of DBs
> > needs some thinking.
> > In DuckDB's case, it's challenging because remote attach over https/s3 is
> > readonly.
> > So even if you stream out IPC in your HTTP server's function (for the /pipe
> > route), you'd need a second HTTP server to receive IPC and write it in...
> > which is fine and still works.
> > But do note that this means accessing the DBs entirely via HTTP, so most of
> > the securty etc will be at the API gateway / HTTP layer.
> >
> >
> >
> > On Monday, March 17th, 2025 at 06:41, Z A <z11...@outlook.com> wrote:
> >
> > Thanks all for the reply and insights! I am going to investigate the grpc
> > passthru feature which is supported by the API gateway product.
> > Btw, I wrote a simple Arrow Flight Server in python, but I couldn't figure
> > out how to call it from grpcurl (all my attempts failed), is that even
> > possible?
> > ________________________________
> > From: Hélder Gregório <helder.grego...@dremio.com>
> > Sent: Saturday, March 15, 2025 8:28 AM
> > To: user@arrow.apache.org <user@arrow.apache.org>
> > Subject: Re: api gateway with arrow flight grpc
> >
> > Hi,
> >
> > I'm not sure about integration with the services you mentioned but assuming
> > some responsibilities of the API Gateway are handling authentication and
> > load balancing, you can implement a flight proxy by creating a service that
> > implements FlightProducer interface.
> > The service can access headers for validation if needed and redirect
> > vectors it receives to the designated flight server using another flight
> > client.
> >
> > Hope it helps
> >
> > On Sat, Mar 15, 2025 at 3:03 AM kekronbekron <kekronbek...@protonmail.com>
> > wrote:
> >
> > Sure -
> >
> > https://www.definite.app/blog/duck-takes-flight
> > https://blobs.duckdb.org/events/duckcon6/rusty-conover-airport-for-duckdb-letting-duckdb-take-apache-arrow-flights.pdf
> >
> >
> >
> >
> > On Saturday, March 15th, 2025 at 01:25, Bryce Mecum <bryceme...@gmail.com>
> > wrote:
> >
> > > Hi kekronbekron, can you share any pointers to the pattern you mention
> > > and where people are talking about it? It sounds like something I
> > > might be interested in tracking.
> > >
> > > On Thu, Mar 13, 2025 at 7:27 PM kekronbekron
> > > kekronbek...@protonmail.com wrote:
> > >
> > > > I'm embarking on exactly this.
> > > > Amusing how this pattern has become "vocal" in the past few weeks,
> > > > though Flight has existed for a few years now.
> > > >
> > > > On Friday, March 14th, 2025 at 03:56, Paul Whalen pgwha...@gmail.com
> > > > wrote:
> > > >
> > > > I haven’t used anything you’ve mentioned, but I have deployed an arrow
> > > > flight server in an istio/envoy service mesh, and seen all the benefits
> > > > of gRPC in that context without any trouble.
> > > >
> > > > I’ve often though it would be fun to write an envoy filter for the
> > > > Flight or Flight SQL protocol, but it isn’t clear to me that there’s
> > > > much demand for it (for comparison, there’s one for Postgres:
> > > > https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_protocols/postgres).
> > > >
> > > > Paul
> > > >
> > > > On Mar 13, 2025, at 5:05 PM, Z A z11...@outlook.com wrote:
> > > >
> > > > Thanks Felipe!
> > > > ________________________________
> > > > From: Felipe Oliveira Carvalho felipe...@gmail.com
> > > > Sent: Thursday, March 13, 2025 12:29 AM
> > > > To: user@arrow.apache.org user@arrow.apache.org
> > > > Subject: Re: api gateway with arrow flight grpc
> > > >
> > > > No, but if these are gRPC proxies they should work.
> > > >
> > > > On Wed, 12 Mar 2025 at 18:13 Z A z11...@outlook.com wrote:
> > > >
> > > > Hi,
> > > > I just subscribed to this mailing list, and apologize if this is a
> > > > silly question.
> > > > Has anyone ever done any integration of API Gateway (i.e. Kong, Tyk,
> > > > KrakenD, etc.) with your own Arrow Flight Server?
> > > > Thanks!
> >
> >
