Instead of changing an existing RPC call, we could add a new optional call for this. And use a GetSqlInfo property so that the FlightSql client application can check if the server supports it?
The response can be oneOf the Arrow ByteString or a FlightInfo object. On Tue, Mar 1, 2022 at 6:01 AM David Li <lidav...@apache.org> wrote: > Feature is part of Schema, so how would that work? The client isn't > providing the server with a schema. (I likely misunderstood something.) > > Flight SQL could indeed use a request field to indicate this to the > server, but I wonder if it's that costly to always return both and let the > client choose how to handle things. > > Does someone want to sketch out a proposal? > > On Mon, Feb 28, 2022, at 14:48, Micah Kornfield wrote: > >> > >> This seems reasonable, however we need to account for existing Flight > >> clients that were written before this. > > > > One way to accomplish this explicitly for FlighSQL could be to add a > field > > which allows clients to say they can handle inline results directly to > the > > requests, if we want to maintain backwards compatibility. Another might > be > > toif the currently unused Feature enum [1] can be used. But since > > FlightSQL is experimental it would be nice to see this feature added > before > > it moves out of experimental. > > > > > > It seems like the server will need to still handle the ticket returned > for > >> getStream() for clients that are unaware of the small result > optimization. > > > > I was thinking the server could have the option of not returning the > ticket > > at all if it returns Arrow data directly via FlightInfo. I think if a > > server does return a ticket then it should be able to handle it. > > > > [1] https://github.com/apache/arrow/blob/master/format/Schema.fbs#L67 > > > > > > On Mon, Feb 28, 2022 at 11:30 AM James Duong > > <jam...@bitquilltech.com.invalid> wrote: > > > >> This seems reasonable, however we need to account for existing Flight > >> clients that were written before this. > >> > >> It seems like the server will need to still handle the ticket returned > for > >> getStream() for clients that are unaware of the small result > optimization. > >> > >> On Mon, Feb 28, 2022 at 11:26 AM David Li <lidav...@apache.org> wrote: > >> > >> > Ah, that makes more sense, that would be a reasonable extension to > Flight > >> > overall. (While we're at it, I think it would help to have an > >> app_metadata > >> > field in FlightInfo as well.) > >> > > >> > On Mon, Feb 28, 2022, at 14:24, Micah Kornfield wrote: > >> > >> > >> > >> But it seems reasonable to add a one-shot query path using DoGet. > >> > > > >> > > > >> > > I was thinking more of adding a bytes field to FlightInfo that could > >> > store > >> > > arrow data. That way GetFlightInfo would be the only RPC necessary > for > >> > > small results when executing a CMD. The client doesn't necessarily > >> know > >> > > whether a query will return large or small results. > >> > > > >> > > On Mon, Feb 28, 2022 at 11:04 AM David Li <lidav...@apache.org> > wrote: > >> > > > >> > >> I think the focus was on large result sets (though I don't recall > this > >> > >> being discussed before) and supporting multi-node setups (hence > >> > >> GetFlightInfo/DoGet are separated). But it seems reasonable to add > a > >> > >> one-shot query path using DoGet. > >> > >> > >> > >> On Mon, Feb 28, 2022, at 13:32, Adam Lippai wrote: > >> > >> > I saw the same. A small, stateless query ability would be nice > >> > >> (connection > >> > >> > open, initialization, query in one message, the resultset in the > >> > response > >> > >> > in one message) > >> > >> > > >> > >> > On Mon, Feb 28, 2022, 13:12 Micah Kornfield < > emkornfi...@gmail.com> > >> > >> wrote: > >> > >> > > >> > >> >> I'm rereviewing the Flight SQL interfaces, and I'm not sure if > I'm > >> > >> missing > >> > >> >> it but is there any optimization for small results? My concern > is > >> > that > >> > >> the > >> > >> >> overhead of the RPCs for the DoGet after executing the query > could > >> > add > >> > >> >> non-trivial latency for smaller results. > >> > >> >> > >> > >> >> Has anybody else thought about this/investigated it? Am I > >> > understanding > >> > >> >> this correctly? > >> > >> >> > >> > >> >> Thanks, > >> > >> >> Micah > >> > >> >> > >> > >> > >> > > >> > >> > >> -- > >> > >> *James Duong* > >> Lead Software Developer > >> Bit Quill Technologies Inc. > >> Direct: +1.604.562.6082 | jam...@bitquilltech.com > >> https://www.bitquilltech.com > >> > >> This email message is for the sole use of the intended recipient(s) and > may > >> contain confidential and privileged information. Any unauthorized > review, > >> use, disclosure, or distribution is prohibited. If you are not the > >> intended recipient, please contact the sender by reply email and destroy > >> all copies of the original message. Thank you. > >> > -- *James Duong* Lead Software Developer Bit Quill Technologies Inc. Direct: +1.604.562.6082 | jam...@bitquilltech.com https://www.bitquilltech.com This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.
