[Rust] Preparing for the DataFusion 6.0.0 Release

I wanted to let people know we are preparing for a DataFusion 6.0.0 release (thanks QP!) There is a PR in DataFusion [1] as well as a draft blog post [2] that we would love feedback and review on. I think this is going to be a pretty awesome release. [1] https://github.com/apache/arrow-datafusio

Re: [DISCUSS] next iteration of flatbuffer structures

I put the draft up here: https://github.com/apache/arrow/pull/11646 Thanks. On Mon, Nov 8, 2021 at 1:57 PM David Li wrote: > Hey Nate, > > Thanks for doing this! Would you be interested in putting that commit up > as a draft PR for discussion? I think we can discuss there. > > I'm not sure anyo

Re: [VOTE] Release Apache Arrow 6.0.1 - RC0

Hi, Thanks for your verification. > dev/release/verify-release-candidate.sh binaries 6.0.1 0 fails > > Reading package lists...E: Invalid archive signature > E: Internal error, could not locate member > control.tar{.zst,.lz4,.gz,.xz,.bz2,.lzma,} > E: Could not read meta data from > /apache-arrow

Re: [VOTE] Release Apache Arrow 6.0.1 - RC0

OK. I withdraw this vote and am going to cut RC1. Thanks, -- kou In "Re: [VOTE] Release Apache Arrow 6.0.1 - RC0" on Mon, 8 Nov 2021 08:01:36 -0500, Neal Richardson wrote: > I think we should include that fix. On the R side we discovered that people > were relying on the old behavior. >

Re: [DISCUSS] next iteration of flatbuffer structures

Hey Nate, Thanks for doing this! Would you be interested in putting that commit up as a draft PR for discussion? I think we can discuss there. I'm not sure anyone is actively working on RLE or other encoding schemes at the moment. -David On Mon, Nov 8, 2021, at 13:19, Nate Bauernfeind wrote:

Re: Question about Arrow Mutable/Immutable Arrays choice

I don't think there is a problem with having "internal" data structures that provide mutation and other capabilities, but when internal data structures are made external (exported to consumers through "public" C++ APIs / namespaces) then immutability is good there (or at least forcing a consumer to

Re: [DISCUSS] next iteration of flatbuffer structures

I've written up the ColumnBag proposal addressing items 1 and 2 on the list. I'm open to any and all feedback/suggestions. I'd be happy to add item 3 (binary metadata) to the proposed change set. Let me know if you want me to whip up the initial suggestion for that version (and whether or not to k

Re: [VOTE] Release Apache Arrow 6.0.1 - RC0

On AlmaLinux 8 Python 3.6.8 openjdk version "1.8.0_312" gcc (GCC) 8.4.1 20200928 ruby 2.7.4p191 Docker version 20.10.10, build b485636 dev/release/verify-release-candidate.sh source 6.0.1 0 passes dev/release/verify-release-candidate.sh wheels 6.0.1 0 Cannot run the verification script directly

Re: [VOTE] Release Apache Arrow 6.0.1 - RC0

I think we should include that fix. On the R side we discovered that people were relying on the old behavior. If we're going to cut a new RC it would be good to include a few other fixes for segfaults that have been discovered: * https://issues.apache.org/jira/browse/ARROW-14519 (merged) * https:

Re: [DISCUSS] Community maintained extension repos for Datafusion

I think a separate non-ASF organization, with a central list of extensions like spark-packages.org sounds like a good idea to me. On Sun, Nov 7, 2021 at 1:34 PM Micah Kornfield wrote: > I'll preface this with not being an expert on these matters but this is my > impression. > > > > Therefore, I

Re: [VOTE] Release Apache Arrow 6.0.1 - RC0

Although causing more delay for the release, I would also vote for including Weston's PR. Otherwise it would be very unfortunate that users can't preserve the existing behaviour (of 5.0). Joris On Sun, 7 Nov 2021 at 22:17, Sutou Kouhei wrote: > Hi, > > Python developers, what do you think about