Hi David, > I see the original thread mentioned Flight support, do you think it'd > be possible to support Flight natively? Or conversely, maybe this > could be a candidate for a new Flight "backend" as has been discussed.
Right now our main priority is addressing the caveats I mentioned above. After that I would like to see if we can get some of the client side optimizations Flight uses either into gRPC directly or perhaps specialized clients. Given my current backlog, I don't think these will happen anytime soon. In my opinion (this is really above my pay-grade) native flight support probably hinges on two things: 1. Customer demand for it. 2. The Flight APIs would likely need to conform to the Cloud API design guidelines [1]. I haven't looked closely enough to see if there are any incompatibilities between the two. I don't recall seeing the discussion on "new flight backends", could you provide a pointer? This might be a shorter path for support. I'd also like to make an adapter for the C++ Datasets API, but again, given my current backlog it will take a while for this to happen. Thanks, Micah [1] https://cloud.google.com/apis/design/ On Sat, Jul 27, 2019 at 6:17 AM David Li <li.david...@gmail.com> wrote: > This is super awesome, thanks for sharing! > > I see the original thread mentioned Flight support, do you think it'd > be possible to support Flight natively? Or conversely, maybe this > could be a candidate for a new Flight "backend" as has been discussed. > > Best, > David > > On 7/26/19, Micah Kornfield <emkornfi...@gmail.com> wrote: > > Hi Arrow Dev, > > As a follow-up to an old thread [1] on working with BigQuery and Arrow. I > > just wanted to share some work that Brian Hulette and I helped out with. > > > > I'm happy to announce there is now preliminary support for reading Arrow > > data in the BigQuery Storage API [1]. Python library support is > available > > in the latest release of google-cloud-bigquery-storage [2][3]. > > > > Caveats: > > - Small cached tables are not supported (same with Avro) > > - Row filters aren't supported yet. > > > > Cheers, > > Micah > > > > [1] > > > https://lists.apache.org/thread.html/6d374dc6c948d3e84b1f0feda1d48eddf905a99c0ef569d46af7f7af@%3Cdev.arrow.apache.org%3E > > [2] https://cloud.google.com/bigquery/docs/reference/storage/ > > [3] https://pypi.org/project/google-cloud-bigquery-storage/ > > [4] > > > https://googleapis.github.io/google-cloud-python/latest/bigquery_storage/gapic/v1beta1/reader.html#google.cloud.bigquery_storage_v1beta1.reader.ReadRowsIterable.to_arrow > > >
