> > That’s awesome!! It’s pretty surreal to request a feature from google and > have it built out.
I hope this is beneficial customers in general. Thank you for filing the request. If I'm reading the code correctly looks like you are transporting the > IPC payload in the protobuf format of the bigquery storage API > > https://github.com/googleapis/google-cloud-python/blob/3d324389b92d43e52486f0fe2aca8b41e950640c/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow.proto Yes, the data is returned in a oneof [1] on a ReadRowsResponse. One needs to specify the data format when creating the read session [2]. [1] https://github.com/googleapis/google-cloud-python/blob/3d324389b92d43e52486f0fe2aca8b41e950640c/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage.proto#L294 [2] https://github.com/googleapis/google-cloud-python/blob/3d324389b92d43e52486f0fe2aca8b41e950640c/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/storage.proto#L182 - Micah On Sat, Jul 27, 2019 at 1:33 PM Wes McKinney <wesmck...@gmail.com> wrote: > Very nice! > > If I'm reading the code correctly looks like you are transporting the > IPC payload in the protobuf format of the bigquery storage API > > > https://github.com/googleapis/google-cloud-python/blob/3d324389b92d43e52486f0fe2aca8b41e950640c/bigquery_storage/google/cloud/bigquery_storage_v1beta1/proto/arrow.proto > > Seems very reasonable. Glad you were able to pull this off! > > On Sat, Jul 27, 2019 at 2:55 PM Jonathan Chiang <chiang...@gmail.com> > wrote: > > > > Hi Micah, > > > > That’s awesome!! It’s pretty surreal to request a feature from google > and have it built out. > > > > Thanks, > > Jonathan > > > > > On Jul 26, 2019, at 8:43 PM, Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > > > Hi Arrow Dev, > > > As a follow-up to an old thread [1] on working with BigQuery and > Arrow. I > > > just wanted to share some work that Brian Hulette and I helped out > with. > > > > > > I'm happy to announce there is now preliminary support for reading > Arrow > > > data in the BigQuery Storage API [1]. Python library support is > available > > > in the latest release of google-cloud-bigquery-storage [2][3]. > > > > > > Caveats: > > > - Small cached tables are not supported (same with Avro) > > > - Row filters aren't supported yet. > > > > > > Cheers, > > > Micah > > > > > > [1] > > > > https://lists.apache.org/thread.html/6d374dc6c948d3e84b1f0feda1d48eddf905a99c0ef569d46af7f7af@%3Cdev.arrow.apache.org%3E > > > [2] https://cloud.google.com/bigquery/docs/reference/storage/ > > > [3] https://pypi.org/project/google-cloud-bigquery-storage/ > > > [4] > > > > https://googleapis.github.io/google-cloud-python/latest/bigquery_storage/gapic/v1beta1/reader.html#google.cloud.bigquery_storage_v1beta1.reader.ReadRowsIterable.to_arrow >
