Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-08 Thread Wes McKinney
I'm won't be around much the next couple weeks to help, but this change should be voted on soon so that the implementations can get refactored sooner rather than later On Thu, Apr 4, 2019 at 2:59 PM Wes McKinney wrote: > > hi Brian, > > I would guess that ListFlights would mainly return named dat

Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-04 Thread Wes McKinney
hi Brian, I would guess that ListFlights would mainly return named datasets having a particular PATH per https://github.com/apache/arrow/blob/master/format/Flight.proto#L176 Presumably if a server provides support for the CMD-type FlightDescriptor, that they would have agreed on a .proto or othe

Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-04 Thread Bryan Cutler
This sounds good to me, but if the FlightDescriptors are opaque commands, e.g. protobufs, will the client have to parse a bunch of unknown commands in order to find the right one? On Wed, Apr 3, 2019 at 9:20 AM Jacques Nadeau wrote: > > > > Can you explain what you call "splits"? > > > > Per Wes

Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-03 Thread Jacques Nadeau
> > Can you explain what you call "splits"? > Per Wes's comments, FlightItineraries inside FlightGetInfo. also is it possible a service have tons of flights? Yes > if so, some kind of > pagination need to be done here? Criteria and the stream interface should be sufficient. We need to work

Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-03 Thread ming zhang
imho, list just show what a service has. does not need to provide detail information about each one. we already have a separate method to fetch detail about a flight. optionally we could change GetFlightInfo(FlightDescriptor) returns (FlightGetInfo) {} to input a stream so someone could fetch a bat

Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-03 Thread Wes McKinney
hi Jacques, I agree with you -- I had this concern also during implementation that the query plans have to be generated both in ListFlights and GetFlightInfo. Antoine -- I think "splits" here means the pieces of a distributed dataset. So if a Flight is spread across multiple hosts, then when you

Re: [DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-03 Thread Antoine Pitrou
On Wed, 3 Apr 2019 08:01:25 -0700 Jacques Nadeau wrote: > Right now, the ListFlights method returns a stream of FlightGetInfo (to be > renamed FlightInfo). This actually turns out to be quite expensive in many > cases since splits have to be generated. I'd like to propose changing this > method to

[DISCUSS] Change Flight ListFlights return value to stream of FlightDescriptor

2019-04-03 Thread Jacques Nadeau
Right now, the ListFlights method returns a stream of FlightGetInfo (to be renamed FlightInfo). This actually turns out to be quite expensive in many cases since splits have to be generated. I'd like to propose changing this method to return a stream of FlightDescriptors instead. What do people thi