Forthcoming nanoarrow 0.7.0 release process

2025-05-30 Thread Dewey Dunnington
Hi all, I'd like to kick off a release for nanoarrow! It's been a few months and we've got some great features and bugfixes merged in. I'll kick off the pre-release checklist [1] on Monday and hope to create a release candidate early in the week unless any blockers come up here or in the issues [

Re: [VOTE][RUST] Release Apache Arrow Rust 55.1.0 RC1

2025-05-13 Thread Dewey Dunnington
+1 (binding)! Verified on MacOS 15.3 (M4) On Mon, May 12, 2025 at 8:40 PM Sundy Li wrote: > +1 (non-binding) > > Verified on Arch Linux x86_64 > > > On 2025/05/12 02:01:30 Adam Reeve wrote: > > +1 (non-binding) > > > > Verified on x64 Fedora 42 > > > > On Sat, 10 May 2025 at 09:23, L. C. Hsieh

Re: [DISCUSS] Arrow Variant Extension Type

2025-05-12 Thread Dewey Dunnington
> I think we should make clear that this extension type is for > transporting Parquet Variants. If we were to design a Variant type > specifically for Arrow, it would probably look a bit different That's a great point...there are definitely advantages to both: keeping the spec identical to Parquet

Re: [VOTE][RUST] Release Apache Arrow Rust Object Store 0.12.1 RC1

2025-05-11 Thread Dewey Dunnington
+1 (binding)! I ran dev/release/verify-release-candidate.sh 55.1.0 1 on MacOS 15.4 (M4) Cheers! -dewey On Fri, May 9, 2025 at 4:42 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M4 Mac. > > Thanks Andrew. > > On Fri, May 9, 2025 at 1:34 PM Andrew Lamb wrote: > > > > Hi, > > > > I would

Re: [DISCUSS] Arrow Variant Extension Type

2025-05-11 Thread Dewey Dunnington
Thanks for pursuing this! This is one of the questions I had reviewing the beginnings of Variant support in Parquet C++ and Go. I would love to see this as a canonical extension type, both for its utility as a type and as further incentive to strengthen the extension system. Cheers, -dewey On Th

Re: [VOTE][Go] Release Apache Arrow Go 18.3.0 RC0

2025-05-07 Thread Dewey Dunnington
+1 (binding) I ran dev/release/verify_rc.sh 18.3.0 0 on MacOS 15.4 (M4) On Wed, May 7, 2025 at 2:13 AM Saurabh Singh wrote: > +1 (Non-binding) > > I ran the following on Ubuntu 24.10/amd64: > > dev/release/verify_rc.sh 18.3.0 0 > > with: > > * go version go1.23.7 linux/amd64 > > Thanks, > S

Re: [VOTE] Release Apache Arrow ADBC 18 - RC0

2025-05-02 Thread Dewey Dunnington
+1 (binding)! I ran USE_CONDA=1 ./verify-release-candidate.sh on MacOS 15.3 (M4) On Fri, May 2, 2025 at 6:03 AM David Li wrote: > Hello, > > I would like to propose the following release candidate (RC0) of Apache > Arrow ADBC version 18. This is a release consisting of 28 resolved GitHub > issu

Re: [VOTE][Format] Extend Flight Location URI Semantics

2025-05-01 Thread Dewey Dunnington
+1 (binding)! On Thu, May 1, 2025 at 10:42 PM Ian Cook wrote: > +1 (binding) > > Ian > > On Thu, May 1, 2025 at 9:51 PM David Li wrote: > > > +1 (binding) > > > > On Fri, May 2, 2025, at 08:00, Joel Lubinitsky wrote: > > > +1 (non-binding) > > > > > > Thanks for putting this together Matt. > >

Re: [VOTE][RUST] Release Apache Arrow Rust 55.0.0 RC1

2025-04-08 Thread Dewey Dunnington
+1! I ran dev/release/verify-release-candidate.sh 55.0.0 1 on MacOS M4 On Tue, Apr 8, 2025 at 12:39 PM Ed Seidl wrote: > +1 (non-binding) > > Verified on RHEL8 x86_64 > > Thanks Andrew! > Ed > > On 2025/04/08 16:11:57 Raúl Cumplido wrote: > > +1 > > > > I've run the verification script successf

Re: [VOTE] Enable GitHub Discussions for apache/arrow-*

2025-04-05 Thread Dewey Dunnington
+1! On Fri, Mar 21, 2025 at 7:58 AM Neal Richardson wrote: > +1 > > On Fri, Mar 21, 2025 at 5:18 AM Raúl Cumplido wrote: > > > +1 > > > > happy to try it > > > > El vie, 21 mar 2025 a las 7:42, Jacob Quinn () > > escribió: > > > > > +1 > > > > > > On Thu, Mar 20, 2025 at 10:50 PM Sutou Kouhei

Re: [VOTE][RUST] Release Apache Arrow Rust 54.3.1 RC1

2025-03-26 Thread Dewey Dunnington
+1! I ran dev/release/verify-release-candidate.sh 54.3.1 1 on MacOS 15 (M4) On Wed, Mar 26, 2025 at 1:32 PM Matthijs Brobbel wrote: > +1 (non-binding) > Verified on macOS 15.3 (aarch64) > > On 2025/03/26 16:17:14 Andrew Lamb wrote: > > Hi, > > > > I would like to propose a release of Apache Arr

Re: [ANNOUNCE] New Arrow PMC member: Ian Cook

2025-03-20 Thread Dewey Dunnington
Congratulations! On Thu, Mar 20, 2025 at 9:41 AM Bryce Mecum wrote: > Congratulations Ian! > > On Thu, Mar 20, 2025 at 1:05 AM Sutou Kouhei wrote: > > > > The Project Management Committee (PMC) for Apache Arrow has invited > > Ian Cook to become a PMC member and we are pleased to announce > > t

Re: [ANNOUNCE] New Arrow PMC member: Rok Mihevc

2025-03-19 Thread Dewey Dunnington
Congratulations! On Wed, Mar 19, 2025 at 4:11 PM Fokko Driesprong wrote: > Congrats Rok! > > Op wo 19 mrt 2025 om 22:08 schreef Adam Reeve > > > Congratulations Rok! > > > > On Thu, 20 Mar 2025 at 09:35, Krisztián Szűcs > > > wrote: > > > > > Congrats Rok! > > > > > > > On 2025. Mar 19., at 21

Re: [VOTE][RUST] Release Apache Arrow Rust 54.3.0 RC1

2025-03-17 Thread Dewey Dunnington
+1! Verified on MacOS 15.3 (M4). On Mon, Mar 17, 2025 at 5:19 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M4 Mac. > > Thanks Andrew. > > On Mon, Mar 17, 2025 at 2:05 PM Andrew Lamb wrote: > > > > Hi, > > > > I would like to propose a release of Apache Arrow Rust Implementation, > > ve

Re: [ANNOUNCE] New Arrow PMC member: Jacob Wujciak

2025-03-17 Thread Dewey Dunnington
Congratulations! On Mon, Mar 17, 2025 at 12:29 PM Jacob Wujciak wrote: > Thank you everyone! > > Bryce Mecum schrieb am Mo., 17. März 2025, 17:25: > > > Congrats! > > > > On Sun, Mar 16, 2025 at 10:23 PM Sutou Kouhei > wrote: > > > > > > The Project Management Committee (PMC) for Apache Arrow

Re: [VOTE][Go] Release Apache Arrow Go 18.2.0 RC1

2025-03-13 Thread Dewey Dunnington
+1! (binding) I ran dev/release/verify_rc.sh 18.2.0 1 on MaOS 15 On Thu, Mar 13, 2025 at 11:04 AM Saurabh Singh wrote: > +1 (Non-binding) > > I ran `dev/release/verify_rc.sh 18.2.0 1` successfully on Ubuntu 24.10. > > with: > > * go version go1.23.6 linux/amd64 > > > Thanks, > Saurabh > > > O

Re: [VOTE][Go] Release Apache Arrow Go 18.2.0 RC0

2025-03-11 Thread Dewey Dunnington
+1 (binding) I ran dev/release/verify_rc.sh 18.2.0 0 on go version go1.24.1 darwin/arm64 On Tue, Mar 11, 2025 at 3:38 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > Regards > JB > > On Mon, Mar 10, 2025 at 9:00 PM Matt Topol wrote: > > > > Hi, > > > > I would like to propose the followi

Re: [VOTE] Release Apache Arrow ADBC 17 - RC0

2025-03-03 Thread Dewey Dunnington
+1 (binding)! Verified on MacOS 14 (aarch64) with `USE_CONDA=1 ./verify-release-candidate.sh 17 0` On Mon, Mar 3, 2025 at 7:17 PM Bryce Mecum wrote: > +1 (binding) > > Verified on macOS 15 (aarch64) with > `dev/release/verify-release-candidate.sh 17 0` after a few tweaks: > > - Added libpq and

Re: [ANNOUNCE] New Arrow PMC member: Bryce Mecum

2025-02-06 Thread Dewey Dunnington
Congratulations! On Thu, Feb 6, 2025 at 1:56 PM Krisztián Szűcs wrote: > Congrats, Bryce! > > > On 2025. Feb 6., at 14:01, Li Jin wrote: > > > > Congrats! > > > > On Thu, Feb 6, 2025 at 2:52 AM wish maple > wrote: > > > >> Congrats! > >> > >> Best, > >> Xuwei Fu > >> > >> Raúl Cumplido 于2025年

Re: [RESULT][VOTE] Release Apache Arrow ADBC 16 - RC3

2025-01-22 Thread Dewey Dunnington
Apologies for missing the vote! I submitted all the R packages this morning :) [x] Update R packages On Mon, Jan 20, 2025 at 8:31 PM David Li wrote: > Post-release tasks: > > [x] Close the GitHub milestone/project > [x] Add the new release to the Apache Reporter System > [x] Upload source relea

Re: [VOTE][Go] Release Apache Arrow Go 18.1.0 RC0

2025-01-10 Thread Dewey Dunnington
+1 (binding)! Verified on MacOS M1 with dev/release/verify_rc.sh 18.1.0 0 On Fri, Jan 10, 2025 at 9:36 PM Bryce Mecum wrote: > +1 (non-binding) > > Verified on macOS 15.1.1 (aarch64). > > On Thu, Jan 9, 2025 at 12:37 PM Matt Topol wrote: > > > > Hi, > > > > I would like to propose the followin

Re: [VOTE] Release Apache Arrow ADBC 16 - RC1

2025-01-10 Thread Dewey Dunnington
+1! I ran USE_CONDA=1 TEST_APT=0 TEST_YUM=0 ./verify-release-candidate.sh 16 1 On Fri, Jan 10, 2025 at 7:30 AM Raúl Cumplido wrote: > Hi, > > I am not sure whether this is an issue with my environment or a real issue > but the verification is failing consistently for the YUM packages for > Alam

Re: [VOTE] Apache Arrow array representation of statistics

2024-12-24 Thread Dewey Dunnington
+1 (binding)! Thank you for collecting all of our comments on this! Cheers, -dewey On Tue, Dec 24, 2024 at 7:52 AM Andrew Lamb wrote: > +1 (binding) > > I think the proposal is well reasoned, incorporates feedback so far, and > will be generally useful (even within the Rust Arrow/DataFusion ec

Re: [DISCUSS] Arrow array representation of statistics

2024-12-13 Thread Dewey Dunnington
First of all, thank you for driving this proposal! I don't think there's anything particularly bad or wrong about mentioning the C data interface in the title of the document...my initial comment was mostly a reaction to the fact that most of the content of the proposal is describing this schema, w

Re: [VOTE] Statistics through the C data interface

2024-12-05 Thread Dewey Dunnington
With apologies for not reviewing this earlier, I've reviewed it now! I am +0 in its current state just because of the title (the proposal seems to be about abstract arrays and not necessarily the C data interface except for the title). Happy to keep up with reviews to get this merged soon! Cheers

Re: [VOTE][Julia] Release Apache Arrow Julia 2.8.0 RC2

2024-12-05 Thread Dewey Dunnington
+1! (binding) I ran dev/release/verify_rc.sh 2.8.0 2 on MacOS M1. On Thu, Dec 5, 2024 at 8:52 AM Jacob Quinn wrote: > > +1 > > Verified on macos m3 > > -Jacob > > > On Thu, Dec 5, 2024 at 2:46 AM Raúl Cumplido wrote: > > > +1 (binding) > > > > Verified on Ubuntu 24.04 > > > > El jue, 5 dic 2024

CVE-2024-52338: Apache Arrow R package: Arbitrary code execution when loading a malicious data file

2024-11-28 Thread Dewey Dunnington
Severity: critical Affected versions: - Apache Arrow R package 4.0.0 through 16.1.0 Description: Deserialization of untrusted data in IPC and Parquet readers in the Apache Arrow R package versions 4.0.0 through 16.1.0 allows arbitrary code execution. An application is vulnerable if it reads

Re: [RESULT][VOTE] Release Apache Arrow ADBC 15 - RC1

2024-11-22 Thread Dewey Dunnington
> > On Tue, Nov 12, 2024, at 20:38, Raúl Cumplido wrote: > >> +1 (binding) > >> > >> I ran "USE_CONDA=1 ./dev/release/verify-release-candidate.sh 15 1" > >> successfully on Ubuntu 24.04. > >> > >> Regards, > >> Raúl > >> &

Re: [VOTE] Release Apache Arrow ADBC 15 - RC1

2024-11-11 Thread Dewey Dunnington
+1 (binding) I ran USE_CONDA=1 TEST_WHEELS=0 ./verify-release-candidate.sh on Ubuntu 22.04 I wasn't able to verify wheels because of [1], but I don't think this is a blocker for the release since the failure is not reproducible anywhere else and is not related to any wheel (just something about h

Re: [ANNOUNCE] New Arrow PMC member: Curt Hagenlocher

2024-10-30 Thread Dewey Dunnington
Congrats! On Wed, Oct 30, 2024 at 8:38 PM Bryce Mecum wrote: > > Congrats Curt! > > On Wed, Oct 30, 2024 at 2:56 PM Sutou Kouhei wrote: > > > > The Project Management Committee (PMC) for Apache Arrow has > > invited Curt Hagenlocher to become a PMC member and we are > > pleased to announce that

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.4 RC1

2024-10-29 Thread Dewey Dunnington
+1 (binding) Verified on MacOS M1 (Julia 1.10) On Mon, Oct 28, 2024 at 3:38 PM Sutou Kouhei wrote: > > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.7.4. > > This release candidate is based on commit: > 91e271eee38e8b376a069d3b09065cefe32

Re: [VOTE] Add Async C Data Interface

2024-10-24 Thread Dewey Dunnington
Thanks Matt for putting this together! I was initially concerned about the complexity of the proposal; however, it is a difficult interaction to standardize and this proposal is not so complex that it is unimplementable. I am excited to use this to improve our asynchronous database access story in

Re: [VOTE][Go] Release Apache Arrow Go 18.0.0 RC0

2024-10-16 Thread Dewey Dunnington
+1! (binding) I ran `dev/release/verify_rc.sh 18.0.0 0` on MacOS 14.6 (go 1.23.2) Cheers, -dewey On Wed, Oct 16, 2024 at 7:13 PM Sutou Kouhei wrote: > > +1 (binding) > > I ran the following on Debian GNU/Linux sid: > > dev/release/verify_rc.sh 18.0.0 0 > > with: > > * go version go1.23.1 l

[VOTE][RESULT] Release Apache Arrow nanoarrow 0.6.0

2024-10-12 Thread Dewey Dunnington
from SVN [x] Bumped versions on main [1] https://github.com/mesonbuild/wrapdb/pull/1741 [2] https://github.com/apache/arrow-site/pull/545 On Thu, Oct 10, 2024 at 4:35 PM Dewey Dunnington wrote: > > +1 (verified on MacOS Sonoma) > > -dewey > > On Thu, Oct 10, 2024 at 10:24 AM Vibhat

Re: [VOTE] Release Apache Arrow nanoarrow 0.6.0

2024-10-10 Thread Dewey Dunnington
2024 at 12:28 PM Sutou Kouhei > > wrote: > > > > > > > Hi, > > > > > > > > I also looked at this. It seems that my jsonlite was old > > > > (1.8.4). I upgraded jsonlite to 1.8.8. The failure has > > > > gone. Sorry! >

Re: [VOTE] Release Apache Arrow nanoarrow 0.6.0

2024-10-09 Thread Dewey Dunnington
Bryce can fill in more of the details here, but we consulted on this and it seems like the issue was an out-of-date system `tzdata`. Thanks again for the report! -dewey On Wed, Oct 9, 2024 at 3:36 PM Dewey Dunnington wrote: > > Thanks Bryce! > > I couldn't reproduce this in Doc

Re: [VOTE] Release Apache Arrow nanoarrow 0.6.0

2024-10-09 Thread Dewey Dunnington
Thanks Bryce! I couldn't reproduce this in Docker but I suspect the default image has a highly unconfigured locale/timezone info compared to a more usual user environment. I'll look into this this evening. Cheers, -dewey On Wed, Oct 9, 2024 at 1:31 PM Bryce Mecum wrote: > > Hi Dewey, > > After

Re: [VOTE] Release Apache Arrow nanoarrow 0.6.0

2024-10-09 Thread Dewey Dunnington
> > with: > > * Apache Arrow C++ main > * gcc (Debian 13.3.0-2) 13.3.0 > * R version 4.4.1 (2024-06-14) -- "Race for Your Life" > * Python 3.12.4 > > > Thanks, > -- > kou > > > In > "[VOTE] Release Apache Arrow nanoarrow 0.6.0&qu

[VOTE] Release Apache Arrow nanoarrow 0.6.0

2024-10-08 Thread Dewey Dunnington
Hello, I would like to propose the following release candidate (rc0) of Apache Arrow nanoarrow [0] version 0.6.0. This release consists of 114 resolved GitHub issues from 10 contributors [1]. This release candidate is based on commit: 33d2c8b973d8f8f424e02ac92ddeaace2a92f8dd [2] The source relea

[ANNOUNCE] New Arrow committer: Will Ayd

2024-10-01 Thread Dewey Dunnington
On behalf of the Arrow PMC, I'm happy to announce that Will Wyd has accepted an invitation to become a committer on Apache Arrow. Welcome, and thank you for your contributions! -dewey

Re: [RESULT][VOTE] Release Apache Arrow ADBC 14 - RC0

2024-09-06 Thread Dewey Dunnington
Apologies for missing this vote! The updated R packages adbcdrivermanager, adbcpostgresql, and adbcsqlite are now on CRAN. -dewey On Thu, Sep 5, 2024 at 3:12 AM David Li wrote: > > [x] Close the GitHub milestone/project > [x] Add the new release to the Apache Reporter System > [x] Upload source

Re: [VOTE][Format] Bool8 Canonical Extension Type

2024-08-05 Thread Dewey Dunnington
+1 (binding). Thanks for this addition! Looked through the spec and C++/Python PRs. On Mon, Aug 5, 2024 at 11:57 AM Weston Pace wrote: > > +1 (binding) > > Looked through the spec & C++/python PRs. > > On Mon, Aug 5, 2024 at 7:41 AM Ian Cook wrote: > > > +1 (non-binding) > > > > I reviewed the

Re: [VOTE][Format] Opaque canonical extension type

2024-07-24 Thread Dewey Dunnington
+1 (binding)! (Thanks!) On Wed, Jul 24, 2024 at 10:10 AM wish maple wrote: > > +1 (non-binding) > > Checked spec change and C++ impl. > > Best, > Xuwei Fu > > Gang Wu 于2024年7月24日周三 20:51写道: > > > +1 (non-binding) > > > > Checked spec change and C++ impl. > > > > On Wed, Jul 24, 2024 at 6:52 PM J

Re: [DISCUSS] Split Go release process

2024-07-19 Thread Dewey Dunnington
> Extract go/ in apache/arrow to apache/arrow-go like apache/arrow-rs I am not go maintainer and would defer to Matt and Joel on whether or not this is more work from their end, but it does seem like Go users have been forced to change their import paths quite a lot already and would probably be O

Re: [DISCUSS] 8-bit Boolean Canonical Extension Type

2024-07-19 Thread Dewey Dunnington
. > >>>>>> - Numpy will write 0 or 1 to the value buffer to indicate False or > >>>> True. > >>>>>> Importantly, numpy will also understand values outside this range > >> to > >>>> mean > >>>>>> True without re

Re: [DISCUSS] 8-bit Boolean Canonical Extension Type

2024-07-17 Thread Dewey Dunnington
Thank you for this! I have definitely run across the one-byte-per-item bool in numpy, DuckDB, and cudf. I haven't heard any discussion about DuckDB here but I am fairly sure that they represent their boolean type as an int8 as well [1]. > Before the vote, I would like to see verification that this

Re: [DISCUSS] Statistics through the C data interface

2024-06-06 Thread Dewey Dunnington
Thank you for collecting all of our opinions on this! I also agree that (4) is the best option. > Fields: > > | Name | Type | Comments | > ||---| | > | column | utf8 | (2) | The uft8 type would p

Re: [DISCUSS] Migration of parquet-cpp issues to GitHub

2024-05-29 Thread Dewey Dunnington
> INFRA tickets are required before migration. Perhaps this is different for existing repositories, but just a note that it may also be possible by editing .asf.yaml (e.g. [1]) [1] https://github.com/apache/arrow-nanoarrow/blob/81711045e8bb4ded1cb3b5a6fa354b35f18aa4e7/.asf.yaml#L24-L25 On Wed,

Re: [RESULT] Release Apache Arrow nanoarrow 0.5.0

2024-05-29 Thread Dewey Dunnington
:57 PM Dewey Dunnington wrote: > > The vote carries with 4 binding +1s and 3 non-binding +1s. Thank you > everybody for voting! > > There are still a few post-release tasks to complete that I will take > care of this week: > > [x] Closed GitHub milestone > [x] Added rel

[ANNOUNCE] Apache Arrow nanoarrow 0.5.0 Released

2024-05-29 Thread Dewey Dunnington
The Apache Arrow community is pleased to announce the 0.5.0 release of Apache Arrow nanoarrow. This initial release covers 79 resolved issues from 9 contributors[1]. The release is available now from [2], release notes are available at [3], and a blog post highlighting new features and breaking ch

[RESULT] Release Apache Arrow nanoarrow 0.5.0

2024-05-25 Thread Dewey Dunnington
versions on main On Sat, May 25, 2024 at 9:08 PM Dewey Dunnington wrote: > > +1 (binding) > > I ran ./verify-release-candidate.sh 0.5.0 0 on MacOS M1. Also see a > suite of successful verification runs from CI [1] and matrix of Python > wheel builds [2]. > > [1] https:/

Re: [VOTE] Release Apache Arrow nanoarrow 0.5.0

2024-05-25 Thread Dewey Dunnington
nd line on Debian GNU/Linux sid: > > > > > > > > dev/release/verify-release-candidate.sh 0.5.0 0 > > > > > > > > with: > > > > > > > > * Apache Arrow C++ main > > > > * gcc (Debian 13.2.0-23) 13.2.0 > > &g

Re: [DISCUSS] Statistics through the C data interface

2024-05-23 Thread Dewey Dunnington
pendent and can complicate the > > implementation of importers/exporters which would hinder the adoption > > of the C Data Interface. Statistics also bring in security concerns > > that are application-specific. e.g. can an algorithm trust min/max > > stats and risk producing incorr

Re: [VOTE] Release Apache Arrow ADBC 12 - RC4

2024-05-23 Thread Dewey Dunnington
The adbcdrivermanager, adbcsqlite, and adbcpostgresql packages are all updated on CRAN! On Tue, May 21, 2024 at 10:41 PM David Li wrote: > > [x] Close the GitHub milestone/project > [x] Add the new release to the Apache Reporter System > [x] Upload source release artifacts to Subversion > [x] Cre

Re: [DISCUSS] Statistics through the C data interface

2024-05-23 Thread Dewey Dunnington
Thank you for the background! I understand that these statistics are important for query planning; however, I am not sure that I follow why we are constrained to the ArrowSchema to represent them. The examples given seem to going through Python...would it be easier to request statistics at a higher

[VOTE] Release Apache Arrow nanoarrow 0.5.0

2024-05-22 Thread Dewey Dunnington
Hello, I would like to propose the following release candidate (rc0) of Apache Arrow nanoarrow [0] version 0.5.0. This is an initial release consisting of 79 resolved GitHub issues from 9 contributors [1]. This release candidate is based on commit: c5fb10035c17b598e6fd688ad9eb7b874c7c631b [2] Th

Re: [DISCUSS] Statistics through the C data interface

2024-05-22 Thread Dewey Dunnington
I am definitely in favor of adding (or adopting an existing) ABI-stable way to transmit statistics (the one that comes up most frequently for me is just the number of values that are about to show up in an ArrowArrayStream, since the producer often knows this and the consumer often would like to pr

Re: [VOTE] Release Apache Arrow ADBC 12 - RC4

2024-05-17 Thread Dewey Dunnington
+1 (binding) Tested with MacOS M1 using TEST_YUM=0 TEST_APT=0 USE_CONDA=1 ./verify-release-candidate.sh 12 4 On Fri, May 17, 2024 at 9:46 AM Jean-Baptiste Onofré wrote: > > +1 (non binding) > > Testing on MacOS M2. > > Regards > JB > > On Wed, May 15, 2024 at 7:00 AM David Li wrote: > > > > Hel

Re: [ANNOUNCE] New Arrow committer: Dane Pitkin

2024-05-07 Thread Dewey Dunnington
Congrats! On Tue, May 7, 2024 at 11:55 AM Raúl Cumplido wrote: > > Congratulations Dane! > > El mar, 7 may 2024, 16:32, Weston Pace escribió: > > > Congrats Dane! > > > > On Tue, May 7, 2024, 7:30 AM Nic Crane wrote: > > > > > Congrats Dane, well deserved! > > > > > > On Tue, 7 May 2024 at 15:1

Re: [Discuss] Extension types based on canonical extension types?

2024-04-30 Thread Dewey Dunnington
I don't think there is any current barrier to using implementation features of one extension type to help with another. In Python, for example, one might be able to do: class GeoJSONExtensionType(pa.ExtensionType): def __init__(self): self._json_ext = pa.JSONExtensionType() def s

Re: ADBC - OS-level driver manager

2024-04-24 Thread Dewey Dunnington
I definitely see the problem here: we don't currently provide a way for something like a Microsoft Excel or PowerBI or Tableau to use ADBC drivers without bundling all of the ones they want to support or requiring/embedding Python or R. I also see how this is a particular problem for Windows and Ma

Re: [DISCUSSION] New Flags for Arrow C Interface Schema

2024-04-22 Thread Dewey Dunnington
is now. If the ARROW_FLAG_RECORD_BATCH flag is set, then it > should error unless calling ImportRecordBatch. It allows the producer to > provide context as to the source and intention of the structure of the data. > > --Matt > > On Fri, Apr 19, 2024 at 8:24 PM Dewey Dunnington > w

Re: [DISCUSSION] New Flags for Arrow C Interface Schema

2024-04-19 Thread Dewey Dunnington
Thanks for bringing this up! Could you share the motivation where this distinction is important in the context of transfer across the C data interface? The "struct == record batch" concept has always made sense to me because in R, a data.frame can have a column that is also a data.frame and there

Re: [ANNOUNCE] New Arrow committer: Sarah Gilmore

2024-04-11 Thread Dewey Dunnington
Congratulations! On Thu, Apr 11, 2024 at 2:23 PM Alenka Frim wrote: > > Congratulations Sarah! > > On Thu, Apr 11, 2024 at 6:21 PM Ruoxi Sun wrote: > > > Congrats! > > > > *Regards,* > > *Rossi SUN* > > > > > > Weston Pace 于2024年4月12日周五 00:13写道: > > > > > Congratulations! > > > > > > On Thu, Ap

Re: Unsupported/Other Type

2024-04-11 Thread Dewey Dunnington
Depending where your Arrow-encoded data is used, either extension types or generic field metadata are options. We have this problem in the ADBC Postgres driver, where we can convert *most* Postgres types to an Arrow type but there are some others where we can't or don't know or don't implement a co

Re: Arrow community meeting April 10 at 16:00 UTC

2024-04-10 Thread Dewey Dunnington
Hi Ian, I'll be attending and I'm happy to run the meeting. Cheers! -dewey On Tue, Apr 9, 2024 at 9:41 PM Ian Cook wrote: > > Our next biweekly Arrow community meeting is tomorrow at 16:00 UTC / 12:00 > EDT. > > I will not be able to attend tomorrow. Could someone please volunteer to > lead th

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-03 Thread Dewey Dunnington
Thank you Jacob for bringing this up! I am also in favor of decoupling versions (provided that the release managers are also in favor of this, since their time is required to implement this and because the ongoing consequences of separate releases disproportionately affects them). Part of the vote

Re: [VOTE] Release Apache Arrow ADBC 0.11.0 - RC0

2024-03-28 Thread Dewey Dunnington
+1! I ran: export DOCKER_DEFAULT_PLATFORM=linux/amd64 USE_CONDA=1 dev/release/verify-release-candidate.sh 0.11.0 0 Matt - could you open an issue? The R package is not supposed to run those tests unless some very specific environment variables are defined in ~/.Renviron. On Thu, Mar 28, 2024 at

Re: [ANNOUNCE] New Arrow committer: Bryce Mecum

2024-03-19 Thread Dewey Dunnington
Congratulations Bryce! And thank you! On Mon, Mar 18, 2024 at 2:16 PM Wes McKinney wrote: > > Congrats! > > On Mon, Mar 18, 2024 at 12:15 PM James Duong > wrote: > > > Congratulations Bryce! > > > > From: Dane Pitkin > > Date: Monday, March 18, 2024 at 7:28 AM > > To: dev@arrow.apache.org > >

Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project

2024-03-02 Thread Dewey Dunnington
+1 (binding) On Sat, Mar 2, 2024 at 8:08 AM vin jake wrote: > > +1 (binding) > > On Fri, Mar 1, 2024 at 7:33 PM Andrew Lamb wrote: > > > Hello, > > > > As we have discussed[1][2] I would like to vote on the proposal to > > create a new Apache Top Level Project for DataFusion. The text of the > >

Re: [RESULT][VOTE] Release Apache Arrow ADBC 0.10.0 - RC1

2024-02-23 Thread Dewey Dunnington
21, 2024, at 19:02, Dane Pitkin wrote: > >> +1 (non-binding) > >> > >> Verified on Mac M1 using conda. > >> > >> On Tue, Feb 20, 2024 at 11:27 PM Dewey Dunnington > >> wrote: > >> > >>> +1! > >>> > >>> I ran

Re: [VOTE] Release Apache Arrow ADBC 0.10.0 - RC1

2024-02-20 Thread Dewey Dunnington
+1! I ran USE_CONDA=1 dev/release/verify-release-candidate.sh 0.10.0 1 on MacOS Sonoma (M1). On Tue, Feb 20, 2024 at 9:43 AM Jean-Baptiste Onofré wrote: > > +1 (non binding) > > I quickly tested on MacOS arm64. > > Regards > JB > > On Sun, Feb 18, 2024 at 9:47 PM David Li wrote: > > > > Hello,

Re: [ANNOUNCE] Apache Arrow nanoarrow 0.4.0 Released

2024-02-16 Thread Dewey Dunnington
Thanks for the suggestion! I opened up a PR to update that language [1]. Cheers! -dewey [1] https://github.com/apache/arrow-nanoarrow/pull/389 On Mon, Feb 12, 2024 at 2:57 PM Antoine Pitrou wrote: > > > Hi Dewey, > > Le 12/02/2024 à 15:01, Dewey Dunnington a écrit : > > A

Re: [RESULT] Release Apache Arrow nanoarrow 0.4.0 - RC0

2024-02-12 Thread Dewey Dunnington
] Release blog post [x] Sent announcement to annou...@apache.org [x] Removed old artifacts from SVN [x] Bumped versions on main On Thu, Feb 1, 2024 at 3:21 PM Dewey Dunnington wrote: > > With 4 binding +1 and 1 non-binding +1, the vote carries! > > If somebody is up for reviewing the

[ANNOUNCE] Apache Arrow nanoarrow 0.4.0 Released

2024-02-12 Thread Dewey Dunnington
The Apache Arrow community is pleased to announce the 0.4.0 release of Apache Arrow nanoarrow. This initial release covers 44 resolved issues from 5 contributors[1]. The release is available now from [2], release notes are available at [3], and a blot post documenting new contributions is availabl

[RESULT] Release Apache Arrow nanoarrow 0.4.0 - RC0

2024-02-01 Thread Dewey Dunnington
> > * Apache Arrow C++ main > * gcc (Debian 13.2.0-9) 13.2.0 > * R version 4.3.2 (2023-10-31) -- "Eye Holes" > * Python 3.11.7 > > Thanks, > -- > kou > > In > "[VOTE] Release Apache Arrow nanoarrow 0.4.0 - RC0" on Mon, 29 Jan 2024 > 1

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.1 RC1

2024-01-31 Thread Dewey Dunnington
+1 Tested on MacOS Sonoma (aarch64). I ran export PATH="/Applications/Julia-1.9.app/Contents/Resources/julia/bin:${PATH}" && dev/release/verify_rc.sh 2.7.1 1 On Wed, Jan 31, 2024 at 2:01 PM Jacob Quinn wrote: > > +1, tested on macos. > > -Jacob > > On Wed, Jan 31, 2024 at 10:11 AM Ben Baumgold

Re: [DISCUSS] Status and future of @ApacheArrow Twitter account

2024-01-30 Thread Dewey Dunnington
I also find it a useful tool to follow other projects...there may be a good replacement for it at some point but in the meantime I would love to see releases + blog posts tweeted (or retweeted by) the official account. -dewey On Tue, Jan 30, 2024 at 6:01 AM Raúl Cumplido wrote: > > El lun, 29 en

Re: [VOTE] Release Apache Arrow nanoarrow 0.4.0 - RC0

2024-01-30 Thread Dewey Dunnington
l mar, 30 ene 2024 a las 0:30, David Li () escribió: > > > > +1 (binding) > > > > Tested on Debian Linux 'bookworm' > > > > On Mon, Jan 29, 2024, at 10:45, Dane Pitkin wrote: > > > +1 (non-binding) > > > > > > Verified on MacOS

[VOTE] Release Apache Arrow nanoarrow 0.4.0 - RC0

2024-01-29 Thread Dewey Dunnington
Hello, I would like to propose the following release candidate (rc0) of Apache Arrow nanoarrow [0] version 0.4.0. This release consists of 46 resolved GitHub issues from 5 contributors [1]. This release candidate is based on commit: 3f83f4c48959f7a51053074672b7a330888385b1 [2] The source release

Re: [VOTE] Release Apache Arrow ADBC 0.9.0 - RC0

2024-01-04 Thread Dewey Dunnington
+1 I ran: export DOCKER_DEFAULT_PLATFORM=linux/amd64 && USE_CONDA=1 dev/release/verify-release-candidate.sh 0.9.0 0 ...on MacOS M1 Ventura On Thu, Jan 4, 2024 at 9:47 AM Jean-Baptiste Onofré wrote: > > +1 (non binding) > > I checked: > - LICENSE is OK but maybe worth to keep only LICENSE.txt at

Re: [VOTE] Release Apache Arrow 14.0.2 - RC3

2023-12-15 Thread Dewey Dunnington
+1 I ran TEST_DEFAULT=0 TEST_CPP=1 dev/release/verify-release-candidate.sh 14.0.2 3 on MacOS M1. I do get one failing test (gandiva-internals-test) but this has failed for me for the last three versions. Note that the R bindings will have to patch the static libraries we host for convenience inst

Re: [DISCUSS] Semantics of extension types

2023-12-15 Thread Dewey Dunnington
I also like these equivalence traits...in addition to being easy for extension type authors to specify when registering an extension type in Arrow C++, implementations that allow registration like pyarrow and arrow/R would be able to specify them easily, whereas implementing methods, compute functi

Re: [DISCUSS] Semantics of extension types

2023-12-13 Thread Dewey Dunnington
Thank you for opening the discussion here and opening it up! I agree that attaching semantics as metadata and/or documenting them in a central repository is an unreasonable burden to put on extension type authors and Arrow implementations in general. I also agree that operations other than filter

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.0 RC1

2023-12-08 Thread Dewey Dunnington
+1 I ran export PATH="/Applications/Julia-1.9.app/Contents/Resources/julia/bin:$PATH" dev/release/verify_rc.sh 2.7.0 1 ...on MacOS M1 Ventura On Tue, Dec 5, 2023 at 4:38 PM Sutou Kouhei wrote: > > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia ver

Re: [ANNOUNCE] New Arrow committer: Felipe Oliveira Carvalho

2023-12-07 Thread Dewey Dunnington
Congrats! On Thu, Dec 7, 2023 at 4:28 PM Andrew Lamb wrote: > > Congratulations! > > On Thu, Dec 7, 2023 at 3:09 PM Kevin Gurney > wrote: > > > Congratulations, Felipe! > > > > From: Daniël Heres > > Sent: Thursday, December 7, 2023 2:59 PM > > To: dev@arrow.apa

Re: [DISCUSS] Protocol for exchanging Arrow data over REST APIs

2023-11-21 Thread Dewey Dunnington
I also think a set of best practices for Arrow over HTTP would be a valuable resource for the community...even if it never becomes a specification of its own, it will be beneficial for API developers and consumers of those APIs to have a place to look to understand how Arrow can help improve throug

Re: [ANNOUNCE] New Arrow PMC member: Raúl Cumplido

2023-11-13 Thread Dewey Dunnington
Congrats, Raùl! On Mon, Nov 13, 2023 at 3:54 PM Dane Pitkin wrote: > > Congrats, Raul! > > On Mon, Nov 13, 2023 at 2:45 PM Kevin Gurney > wrote: > > > Congratulations, Raúl! > > > > > > From: Nic Crane > > Sent: Monday, November 13, 2023 2:31 PM > > To: dev@arro

Re: [DISCUSS][MATLAB] Proposal for incremental point releases of the MATLAB interface

2023-11-07 Thread Dewey Dunnington
For argument's sake, I might suggest that the process you described in your initial note would probably work best in another repo: you would be able to iterate faster and release/version at your own pace. The flexibility you get from moving to a separate repo comes at the cost of extra responsibili

Re: [VOTE] Release Apache Arrow ADBC 0.8.0 - RC0

2023-11-07 Thread Dewey Dunnington
+1! I ran: TEST_APT=0 TEST_YUM=0 USE_CONDA=1 dev/release/verify-release-candidate.sh 0.8.0 0 On Fri, Nov 3, 2023 at 12:18 PM David Li wrote: > > Hello, > > I would like to propose the following release candidate (RC0) of Apache Arrow > ADBC version 0.8.0. This is a release consisting of 42 reso

Re: [DISCUSS][Format] C data interface for Utf8View

2023-10-29 Thread Dewey Dunnington
In the absence of a general solution to the C data interface omitting buffer sizes, I think the original proposal is the best way forward...this is the first type to be added whose buffer sizes cannot be calculated without looping over every element of the array; the buffer sizes are needed to effi

Re: [DISCUSS][Format] C data interface for Utf8View

2023-10-26 Thread Dewey Dunnington
s constantly omitting buffer sizes and consumers constantly recalculating them. On Thu, Oct 26, 2023 at 4:35 PM Dewey Dunnington wrote: > > I'm afraid I've derailed the discussion into solving a bigger problem > than strictly necessary. I don't think this is the time

Re: [DISCUSS][Format] C data interface for Utf8View

2023-10-26 Thread Dewey Dunnington
I'm afraid I've derailed the discussion into solving a bigger problem than strictly necessary. I don't think this is the time to solve the general problem of the C data interface having no way to communicate buffer sizes, particularly since there's no immediate agreement on its utility or implement

Re: [DISCUSS][Format] C data interface for Utf8View

2023-10-26 Thread Dewey Dunnington
to another device?) I don't think there's any barrier to accessing the content of all the array elements but I could be mistaken. On Thu, Oct 26, 2023 at 1:04 PM Antoine Pitrou wrote: > > > Le 26/10/2023 à 17:45, Dewey Dunnington a écrit : > > The lack of buffer sizes is some

Re: [DISCUSS][Format] C data interface for Utf8View

2023-10-26 Thread Dewey Dunnington
Ben kindly explained to me offline that the need for the buffer sizes is because when Arrow C++ imports an Array it creates Buffer class wrappers around the imported pointers. Arrow C++ does not have a notion of a buffer of unknown size to my knowledge, which leaves two undesirable alternatives: (1

Re: [VOTE][Format] C data interface format strings for Utf8View and BinaryView

2023-10-18 Thread Dewey Dunnington
+1! On Wed, Oct 18, 2023 at 2:14 PM Matt Topol wrote: > > +1 > > On Wed, Oct 18, 2023 at 1:05 PM Antoine Pitrou wrote: > > > +1 > > > > Le 18/10/2023 à 19:02, Benjamin Kietzman a écrit : > > > Hello all, > > > > > > I propose "vu" and "vz" as format strings for the Utf8View and > > > BinaryView

Re: Apache Arrow file format

2023-10-18 Thread Dewey Dunnington
Plenty of opinions here already, but I happen to think that IPC streams and/or Arrow File/Feather are wildly underutilized. For the use-case where you're mostly just going to read an entire file into R or Python it's a bit faster (and far superior to a CSV or pickling or .rds files in R). > you're

Re: [ANNOUNCE] New Arrow PMC member: Jonathan Keane

2023-10-15 Thread Dewey Dunnington
Congrats, Jon! On Sun, Oct 15, 2023 at 7:53 AM Nic Crane wrote: > > Congrats Jon! > > On Sun, 15 Oct 2023, 05:52 Jacob Wujciak-Jens, > wrote: > > > Congratulations 🎉! > > > > Raúl Cumplido schrieb am So., 15. Okt. 2023, > > 00:58: > > > > > Congratulations Jon! > > > > > > El dom, 15 oct 2023,

Re: [DISCUSS][Swift] repo for swift similar to arrow-rs

2023-10-10 Thread Dewey Dunnington
Hi Alva, I would encourage you to do whatever will make life more pleasant for you and other contributors to the Swift Arrow implementation. I have found development of an Arrow subproject (nanoarrow) in a separate repository very pleasant. While I don't run integration tests there, it's not becau

Re: [Vote][Format] (new proposal) C data interface format string for ListView and LargeListView arrays

2023-10-06 Thread Dewey Dunnington
+1! On Fri, Oct 6, 2023, 8:03 PM Matt Topol wrote: > +1 > > On Fri, Oct 6, 2023, 6:55 PM Benjamin Kietzman > wrote: > > > +1 > > > > On Fri, Oct 6, 2023, 17:27 Felipe Oliveira Carvalho > > > wrote: > > > > > Hello, > > > > > > I'm writing to propose "+vl" and "+vL" as format strings for list-v

  1   2   >