> > If the current gRPC stub definitions are reasonably stable (in your > opinion), I might try implementing support.
I would guess that is relatively stable, but I don't think I can make any guarantees (as far as I know there are no guarantees made between beta and GA API versions). So while I would love to see this done, it might pay to wait for GA to get some level of stability guarantees. On Mon, Jul 29, 2019 at 6:06 AM David Li <li.david...@gmail.com> wrote: > Hey Micah, > > There hasn't really been formal discussions of Flight "backends", but > there has been some talk about supporting protocols besides gRPC > (which is why the implementation tries to abstract away from gRPC). So > it might be interesting to treat this as another "protocol" in Flight > clients that can only read from BigQuery. > > If the current gRPC stub definitions are reasonably stable (in your > opinion), I might try implementing support. That might get reasonable > performance still, especially in Python (where I've found that a lot > of performance is lost copying messages into/out of CPython to work > with Protobuf & gRPC - presumably grpc-c++ wouldn't do that and/or we > could do the read optimizations ourselves). > > Best, > David > > On 7/27/19, Micah Kornfield <emkornfi...@gmail.com> wrote: > > Hi David, > > > >> I see the original thread mentioned Flight support, do you think it'd > >> be possible to support Flight natively? Or conversely, maybe this > >> could be a candidate for a new Flight "backend" as has been discussed. > > > > Right now our main priority is addressing the caveats I mentioned above. > > After that I would like to see if we can get some of the client side > > optimizations Flight uses either into gRPC directly or perhaps > specialized > > clients. Given my current backlog, I don't think these will happen > anytime > > soon. > > > > In my opinion (this is really above my pay-grade) native flight support > > probably hinges on two things: > > 1. Customer demand for it. > > 2. The Flight APIs would likely need to conform to the Cloud API design > > guidelines [1]. I haven't looked closely enough to see if there are any > > incompatibilities between the two. > > > > I don't recall seeing the discussion on "new flight backends", could you > > provide a pointer? This might be a shorter path for support. I'd also > > like to make an adapter for the C++ Datasets API, but again, given my > > current backlog it will take a while for this to happen. > > > > Thanks, > > Micah > > > > [1] https://cloud.google.com/apis/design/ > > > > On Sat, Jul 27, 2019 at 6:17 AM David Li <li.david...@gmail.com> wrote: > > > >> This is super awesome, thanks for sharing! > >> > >> I see the original thread mentioned Flight support, do you think it'd > >> be possible to support Flight natively? Or conversely, maybe this > >> could be a candidate for a new Flight "backend" as has been discussed. > >> > >> Best, > >> David > >> > >> On 7/26/19, Micah Kornfield <emkornfi...@gmail.com> wrote: > >> > Hi Arrow Dev, > >> > As a follow-up to an old thread [1] on working with BigQuery and > Arrow. > >> > I > >> > just wanted to share some work that Brian Hulette and I helped out > >> > with. > >> > > >> > I'm happy to announce there is now preliminary support for reading > >> > Arrow > >> > data in the BigQuery Storage API [1]. Python library support is > >> available > >> > in the latest release of google-cloud-bigquery-storage [2][3]. > >> > > >> > Caveats: > >> > - Small cached tables are not supported (same with Avro) > >> > - Row filters aren't supported yet. > >> > > >> > Cheers, > >> > Micah > >> > > >> > [1] > >> > > >> > https://lists.apache.org/thread.html/6d374dc6c948d3e84b1f0feda1d48eddf905a99c0ef569d46af7f7af@%3Cdev.arrow.apache.org%3E > >> > [2] https://cloud.google.com/bigquery/docs/reference/storage/ > >> > [3] https://pypi.org/project/google-cloud-bigquery-storage/ > >> > [4] > >> > > >> > https://googleapis.github.io/google-cloud-python/latest/bigquery_storage/gapic/v1beta1/reader.html#google.cloud.bigquery_storage_v1beta1.reader.ReadRowsIterable.to_arrow > >> > > >> > > >
