Can we just add the following field to the FlightDescriptor message:
bool accept_inline_data = 4;
and this one to the FlightInfo message:
FlightData inline_data = 100;
Then new clients can `accept_inline_data` to true (the default being
false if omitted) to signal servers that they can put the data if
`inline_data` if deemed small enough.
(the `accept_inline_data` field could also be used to the Criteria
message)
Alternatively, if the FlightDescriptor expansion looks a bit dirty
(FlightDescriptor being used in other contexts where
`accept_inline_data` makes no sense), we can instead define a new
method:
rpc GetFlightInfoEx(GetFlightInfoRequest) returns (FlightInfo) {}
with:
message GetFlightInfoRequest {
FlightDescriptor flight_descriptor = 1;
bool accept_inline_data = 2;
}
Regards
Antoine.
On Mon, 28 Feb 2022 11:29:12 -0800
James Duong <jam...@bitquilltech.com.INVALID> wrote:
> This seems reasonable, however we need to account for existing Flight
> clients that were written before this.
>
> It seems like the server will need to still handle the ticket returned for
> getStream() for clients that are unaware of the small result optimization.
>
> On Mon, Feb 28, 2022 at 11:26 AM David Li <lidav...@apache.org> wrote:
>
> > Ah, that makes more sense, that would be a reasonable extension to Flight
> > overall. (While we're at it, I think it would help to have an app_metadata
> > field in FlightInfo as well.)
> >
> > On Mon, Feb 28, 2022, at 14:24, Micah Kornfield wrote:
> > >>
> > >> But it seems reasonable to add a one-shot query path using DoGet.
> > >
> > >
> > > I was thinking more of adding a bytes field to FlightInfo that could
> > store
> > > arrow data. That way GetFlightInfo would be the only RPC necessary for
> > > small results when executing a CMD. The client doesn't necessarily know
> > > whether a query will return large or small results.
> > >
> > > On Mon, Feb 28, 2022 at 11:04 AM David Li <lidav...@apache.org> wrote:
> > >
> > >> I think the focus was on large result sets (though I don't recall this
> > >> being discussed before) and supporting multi-node setups (hence
> > >> GetFlightInfo/DoGet are separated). But it seems reasonable to add a
> > >> one-shot query path using DoGet.
> > >>
> > >> On Mon, Feb 28, 2022, at 13:32, Adam Lippai wrote:
> > >> > I saw the same. A small, stateless query ability would be nice
> > >> (connection
> > >> > open, initialization, query in one message, the resultset in the
> > response
> > >> > in one message)
> > >> >
> > >> > On Mon, Feb 28, 2022, 13:12 Micah Kornfield <emkornfi...@gmail.com>
> > >> wrote:
> > >> >
> > >> >> I'm rereviewing the Flight SQL interfaces, and I'm not sure if I'm
> > >> missing
> > >> >> it but is there any optimization for small results? My concern is
> > that
> > >> the
> > >> >> overhead of the RPCs for the DoGet after executing the query could
> > add
> > >> >> non-trivial latency for smaller results.
> > >> >>
> > >> >> Has anybody else thought about this/investigated it? Am I
> > understanding
> > >> >> this correctly?
> > >> >>
> > >> >> Thanks,
> > >> >> Micah
> > >> >>
> > >>
> >
>
>
