Feature is part of Schema, so how would that work? The client isn't providing the server with a schema. (I likely misunderstood something.)
Flight SQL could indeed use a request field to indicate this to the server, but I wonder if it's that costly to always return both and let the client choose how to handle things. Does someone want to sketch out a proposal? On Mon, Feb 28, 2022, at 14:48, Micah Kornfield wrote: >> >> This seems reasonable, however we need to account for existing Flight >> clients that were written before this. > > One way to accomplish this explicitly for FlighSQL could be to add a field > which allows clients to say they can handle inline results directly to the > requests, if we want to maintain backwards compatibility. Another might be > toif the currently unused Feature enum [1] can be used. But since > FlightSQL is experimental it would be nice to see this feature added before > it moves out of experimental. > > > It seems like the server will need to still handle the ticket returned for >> getStream() for clients that are unaware of the small result optimization. > > I was thinking the server could have the option of not returning the ticket > at all if it returns Arrow data directly via FlightInfo. I think if a > server does return a ticket then it should be able to handle it. > > [1] https://github.com/apache/arrow/blob/master/format/Schema.fbs#L67 > > > On Mon, Feb 28, 2022 at 11:30 AM James Duong > <jam...@bitquilltech.com.invalid> wrote: > >> This seems reasonable, however we need to account for existing Flight >> clients that were written before this. >> >> It seems like the server will need to still handle the ticket returned for >> getStream() for clients that are unaware of the small result optimization. >> >> On Mon, Feb 28, 2022 at 11:26 AM David Li <lidav...@apache.org> wrote: >> >> > Ah, that makes more sense, that would be a reasonable extension to Flight >> > overall. (While we're at it, I think it would help to have an >> app_metadata >> > field in FlightInfo as well.) >> > >> > On Mon, Feb 28, 2022, at 14:24, Micah Kornfield wrote: >> > >> >> > >> But it seems reasonable to add a one-shot query path using DoGet. >> > > >> > > >> > > I was thinking more of adding a bytes field to FlightInfo that could >> > store >> > > arrow data. That way GetFlightInfo would be the only RPC necessary for >> > > small results when executing a CMD. The client doesn't necessarily >> know >> > > whether a query will return large or small results. >> > > >> > > On Mon, Feb 28, 2022 at 11:04 AM David Li <lidav...@apache.org> wrote: >> > > >> > >> I think the focus was on large result sets (though I don't recall this >> > >> being discussed before) and supporting multi-node setups (hence >> > >> GetFlightInfo/DoGet are separated). But it seems reasonable to add a >> > >> one-shot query path using DoGet. >> > >> >> > >> On Mon, Feb 28, 2022, at 13:32, Adam Lippai wrote: >> > >> > I saw the same. A small, stateless query ability would be nice >> > >> (connection >> > >> > open, initialization, query in one message, the resultset in the >> > response >> > >> > in one message) >> > >> > >> > >> > On Mon, Feb 28, 2022, 13:12 Micah Kornfield <emkornfi...@gmail.com> >> > >> wrote: >> > >> > >> > >> >> I'm rereviewing the Flight SQL interfaces, and I'm not sure if I'm >> > >> missing >> > >> >> it but is there any optimization for small results? My concern is >> > that >> > >> the >> > >> >> overhead of the RPCs for the DoGet after executing the query could >> > add >> > >> >> non-trivial latency for smaller results. >> > >> >> >> > >> >> Has anybody else thought about this/investigated it? Am I >> > understanding >> > >> >> this correctly? >> > >> >> >> > >> >> Thanks, >> > >> >> Micah >> > >> >> >> > >> >> > >> >> >> -- >> >> *James Duong* >> Lead Software Developer >> Bit Quill Technologies Inc. >> Direct: +1.604.562.6082 | jam...@bitquilltech.com >> https://www.bitquilltech.com >> >> This email message is for the sole use of the intended recipient(s) and may >> contain confidential and privileged information. Any unauthorized review, >> use, disclosure, or distribution is prohibited. If you are not the >> intended recipient, please contact the sender by reply email and destroy >> all copies of the original message. Thank you. >>
