[RFC] Enabling data frames in disaggregated shared memory

2024-04-09 Thread John Groves
This is a request for comments from the Arrow developer community. I’m reaching out to start making the Arrow community aware of work that my team at Micron has recently open-sourced. Because of the Compute Express Link (CXL) standard, sharable disaggregated memory is coming – this is memory share

Arrow community meeting April 10 at 16:00 UTC

2024-04-09 Thread Ian Cook
Our next biweekly Arrow community meeting is tomorrow at 16:00 UTC / 12:00 EDT. I will not be able to attend tomorrow. Could someone please volunteer to lead the meeting and take notes in the Google Doc? The Zoom meeting will work as usual; it does not require a host to start it. Zoom meeting URL

Re: [VOTE] Add new info codes and options keys to ADBC specification

2024-04-09 Thread Joel Lubinitsky
The vote passes with 4 binding, 2 non-binding +1 votes. Thanks everyone! On Mon, Apr 8, 2024 at 2:23 AM Jean-Baptiste Onofré wrote: > +1 (non binding) > > It sounds good to me. Maybe with to update documentation ? > > Thanks > Regards > JB > > On Wed, Apr 3, 2024 at 1:01 PM Joel Lubinitsky wrot

Re: [INFO] Subsets of pyarrow package with pyarrow-core < pyarrow < pyarrow-all on conda

2024-04-09 Thread Ian Cook
Thanks Raúl for taking care to make this minimally disruptive. This might be an inconvenience for some users of PyArrow, but I think the benefits outweigh the inconvenience. Ian On Tue, Apr 9, 2024 at 11:17 AM Raúl Cumplido wrote: > Hi, > > As part of the effort to reduce the footprint of pyarr

[INFO] Subsets of pyarrow package with pyarrow-core < pyarrow < pyarrow-all on conda

2024-04-09 Thread Raúl Cumplido
Hi, As part of the effort to reduce the footprint of pyarrow installations, we have been working on splitting pyarrow into separate packages for conda [1]. Each package will pull different C++ dependencies which will provide different capabilities. This PR [1] will provide 3 packages for pyarrow:

Re: [VOTE] Protocol for Dissociated Arrow IPC Transports

2024-04-09 Thread Matt Topol
Hey JB, The next step for me is going to be converting the document into a markdown version with more prose that we can add to the Arrow documentation site (marked as Experimental of course). > (if I can help in any way :) ) I'll definitely add you as a reviewer when I put the PR up :). But i gu

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-09 Thread Antoine Pitrou
It seems that perhaps this discussion should be rebooted for each individual component, one at a time? Let's start with something simple and obvious, with some frequent contribution activity, such as perhaps Go? Le 09/04/2024 à 14:27, Joris Van den Bossche a écrit : I am also in favor o

Re: [VOTE] Protocol for Dissociated Arrow IPC Transports

2024-04-09 Thread Jean-Baptiste Onofré
Hi Matt, Sorry, I'm late on this vote and in the doc review. I like the idea. Thanks for the very detailed document. Just curious, what's the next step (if I can help in any way :) ) ? Regards JB On Tue, Feb 27, 2024 at 6:35 PM Matt Topol wrote: > > Hey all, > > I'd like to propose a vote for

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-09 Thread Joris Van den Bossche
I am also in favor of this idea in general and in the principle, but (somewhat repeating others) I think we should be aware that this will create _more_ work overall for releasing (refactoring release scripts (at least initially), deciding which version to use for which component, etc), and not les

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-09 Thread Jean-Baptiste Onofré
Hi David, Yeah, maybe my "wording" is not accurate, sorry about that. Core is not correct. Maybe common or another wording is more appropriate. Basically, the idea is to first release a component that it's used by other components. And the release cycle can be "atomic"/decoupled between component

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-09 Thread Ian Cook
I think it is worthwhile to pursue this, but I fear that if we do not proceed very carefully, unforeseen complications could arise, creating even greater work for the release managers. > In general I think that this is not something we neither need to nor want to implement from 0 to 100. > Increme

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-09 Thread David Li
Java has JNI parts, but I think they do not necessarily need to release at the same time as C++, especially since the JAR bundles the libraries; Java could just pick up the latest version of the C++ library whenever it releases. It would make it harder if the next step is to also decouple the re

Re: [DISCUSS] Versioning and releases for apache/arrow components

2024-04-09 Thread Jean-Baptiste Onofré
Hi, Yeah, to be honest, I was more focused on Java versioning. Maybe, we can "group" Arrow components in two major areas: the "core" libs and the components using the "core" libs. C++ can have its own versioning, and the rest is decoupled from each other but it will depend to C++ release. I thin