Re: [Flight Format] Authentication Redesign

2020-09-01 Thread David Li
Hey James, I won't be able to attend tomorrow, but seeing as I've been commenting so far - the changes as summarized in the presentation (+I think the latest comments in the doc) look good to me, thanks for working through the proposal with us. Best, David On 9/1/20, James Duong wrote: > Hi, >

Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC

2020-09-01 Thread Ryan Nicholson
Thank you to all who have viewed and given feedback in the posted material so far. I am working on responses as well as preparing some material which I would like to share for discussion in the sync call tomorrow. I will post the material here ahead of time. Take care, Ryan Nicholson On Mon,

Re: [Flight Format] Authentication Redesign

2020-09-01 Thread James Duong
Hi, I would like to talk about the changes in the design document and PR in tomorrow's meeting. I've created a slide deck to get the discussion going. https://docs.google.com/presentation/d/1Bcvu0o0SEL21EMazA6O9fVfYZnzyvsQoKKBCN0NIdi0/edit?usp=drivesdk Thanks! On Wed, Aug 26, 2020, 13:12 James

Re: [DISCUSS] Big Endian support in Arrow

2020-09-01 Thread Sutou Kouhei
Hi, In <4cc66ae9-a181-c937-3b4d-0d13cabf6...@python.org> "Re: [DISCUSS] Big Endian support in Arrow" on Tue, 1 Sep 2020 19:53:18 +0200, Antoine Pitrou wrote: > I don't know about debian-java, but when I try e.g. debian-cpp, it fails > with the following error: > > $ ARCH=s390x docker-compos

Arrow sync call September 2 at 12:00 US/Eastern, 16:00 UTC

2020-09-01 Thread Neal Richardson
Hi all, Reminder that our biweekly call is tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterward. Neal

Re: conversion between pyspark.DataFrame and pyarrow.Table

2020-09-01 Thread Bryan Cutler
There isn't a direct conversion to/from Spark, I made https://issues.apache.org/jira/browse/SPARK-29040 a while ago for conversion to Spark from an Arrow table. If possible, make a comment there for your use case which might help get support for it. Bryan On Mon, Aug 31, 2020, 9:12 PM Micah Kornf

Re: [Discuss] C stream interface

2020-09-01 Thread Wes McKinney
Per the other e-mail thread I think this is a good idea and will unblock a number of use cases (for example, a C FFI with no knowledge of pyarrow could consume the record batches produced by a Dataset). I'm not sure it's necessary to formalize it / freeze the ABI right away until we get some more p

Re: [DISCUSS] Big Endian support in Arrow

2020-09-01 Thread Antoine Pitrou
Le 01/09/2020 à 04:47, Sutou Kouhei a écrit : > > We can use "ARCH=s390x docker-compose run debian-java" (or > "ARCH=s390x archery docker run debian-java") to debug on > a local amd64 machine with qemu-user-static. I don't know about debian-java, but when I try e.g. debian-cpp, it fails with th

Re: [Discuss] C stream interface

2020-09-01 Thread Antoine Pitrou
Also, for the record, I've temporarily uploaded a rendered version of the doc here: https://pitrou.net/arrowdevdoc/format/CStreamInterface.html Regards Antoine. Le 01/09/2020 à 17:25, Antoine Pitrou a écrit : > > Hello, > > We are proposing an experimental C stream interface which builds on

[Discuss] C stream interface

2020-09-01 Thread Antoine Pitrou
Hello, We are proposing an experimental C stream interface which builds on the existing C data interface and is defined by a single C struct: struct ArrowArrayStream { // Callbacks providing stream functionality int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out); int (*g

[NIGHTLY] Arrow Build Report for Job nightly-2020-09-01-0

2020-09-01 Thread Crossbow
Arrow Build Report for Job nightly-2020-09-01-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-09-01-0 Failed Tasks: - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-09-01-0-github-test-conda-pyt