Arrow community meeting August 16 at 16:00 UTC

2023-08-16 Thread Ian Cook
Our next biweekly Arrow community meeting is today at 16:00 UTC / 12:00 EDT. Zoom meeting URL: https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09 Meeting ID: 876 4903 3008 Passcode: 958092 Meeting notes will be captured in this Google Doc: https://docs.google.com/document/d/1xrji8

[Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Matt Topol
Hey All, As proposed by Felipe [1] I'm starting a vote on the proposed update to the Format Spec of adding "+r" as the format string for passing Run-End Encoded arrays through the Arrow C Data Interface. A PR containing an update to the C++ Arrow implementation to add support for this format stri

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Ian Cook
+1 (non-binding) On Wed, Aug 16, 2023 at 10:16 AM Matt Topol wrote: > > Hey All, > > As proposed by Felipe [1] I'm starting a vote on the proposed update to the > Format Spec of adding "+r" as the format string for passing Run-End Encoded > arrays through the Arrow C Data Interface. > > A PR cont

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Dewey Dunnington
+1! Looking forward to implementing this in nanoarrow! On Wed, Aug 16, 2023 at 11:18 AM Ian Cook wrote: > > +1 (non-binding) > > On Wed, Aug 16, 2023 at 10:16 AM Matt Topol > wrote: > > > > Hey All, > > > > As proposed by Felipe [1] I'm starting a vote on the proposed update to the > > Format Sp

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread David Li
+1 On Wed, Aug 16, 2023, at 10:21, Dewey Dunnington wrote: > +1! Looking forward to implementing this in nanoarrow! > > On Wed, Aug 16, 2023 at 11:18 AM Ian Cook wrote: >> >> +1 (non-binding) >> >> On Wed, Aug 16, 2023 at 10:16 AM Matt Topol >> wrote: >> > >> > Hey All, >> > >> > As proposed by

[DISCUSS][Arrow] Extension metadata encoding design

2023-08-16 Thread Jeremy Leibs
Hello, I've recently started working with extension types as part of our project and I was surprised to discover that extension types are required to pack all of their own metadata into a single string value of the "ARROW:extension:metadata" key. In turn this then means we have to endure arbitrar

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Antoine Pitrou
+1 from me (binding). It would be nice to get approval from authors of other implementations such as Rust, C#, Javascript... Thanks for doing this! Le 16/08/2023 à 16:16, Matt Topol a écrit : Hey All, As proposed by Felipe [1] I'm starting a vote on the proposed update to the Format Spec

Re: [DISCUSS][Arrow] Extension metadata encoding design

2023-08-16 Thread Antoine Pitrou
Hi Jeremy, A single key makes it easier for generic code to recreate extension types it does not know about. Here is an example in the C++ IPC layer: https://github.com/apache/arrow/blob/641201416c1075edfd05d78b539275065daac31d/cpp/src/arrow/ipc/metadata_internal.cc#L823-L845 Here is simila

Re: [DISCUSS][Arrow] Extension metadata encoding design

2023-08-16 Thread Jeremy Leibs
Thanks for the context, Antoine. However, even in those examples, I don't really see how coercing the metadata to a single string makes much of a difference. I believe the main difference of what I'm proposing would be that the ExtensionType::Deserialize interface: https://github.com/apache/arrow/

Re: [DISCUSS][Arrow] Extension metadata encoding design

2023-08-16 Thread Antoine Pitrou
Hmm, you're right that letting the extension type peek at the entire metadata values would have been another solution. That said, for protocol compatibility reasons, we cannot easily change this anymore. Regards Antoine. Le 16/08/2023 à 17:48, Jeremy Leibs a écrit : Thanks for the con

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Matt Topol
> It would be nice to get approval from authors of other implementations such as Rust, C#, Javascript... I'm hoping that some of them see this and participate in the vote. *crosses fingers* On Wed, Aug 16, 2023 at 11:10 AM Antoine Pitrou wrote: > > +1 from me (binding). > > It would be nice to

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Raphael Taylor-Davies
+1 (binding) On 16/08/2023 16:58, Matt Topol wrote: It would be nice to get approval from authors of other implementations such as Rust, C#, Javascript... I'm hoping that some of them see this and participate in the vote. *crosses fingers* On Wed, Aug 16, 2023 at 11:10 AM Antoine Pitrou wrot

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread Jacob Quinn
+1 (binding) Cheers, -Jacob On Wed, Aug 16, 2023 at 8:16 AM Matt Topol wrote: > Hey All, > > As proposed by Felipe [1] I'm starting a vote on the proposed update to the > Format Spec of adding "+r" as the format string for passing Run-End Encoded > arrays through the Arrow C Data Interface. >

Re: [VOTE] Apache Arrow ADBC (API) 1.1.0

2023-08-16 Thread Antoine Pitrou
+1 (binding) Le 14/08/2023 à 19:39, David Li a écrit : Hello, We have been discussing revisions [1] to the ADBC APIs, which we formerly decided to treat as a specification [2]. These revisions clean up various missing features (e.g. cancellation, error metadata) and better position ADBC t

Re: [DISCUSS][Arrow] Extension metadata encoding design

2023-08-16 Thread Jeremy Leibs
I realize it's a not-insignificant change, and I'm not (yet) proposing such a change without more discussion and thought into the consequences. But, I don't think this would actually break any protocol, so I don't want to prematurely preclude this as a possible future direction. My understanding

Re: [Vote][Format] C Data Interface Format string for REE

2023-08-16 Thread L. C. Hsieh
+1 (binding) On Wed, Aug 16, 2023 at 9:05 AM Jacob Quinn wrote: > > +1 (binding) > > Cheers, > > -Jacob > > On Wed, Aug 16, 2023 at 8:16 AM Matt Topol > wrote: > > > Hey All, > > > > As proposed by Felipe [1] I'm starting a vote on the proposed update to the > > Format Spec of adding "+r" as the