Re: [VOTE][Julia] Release Apache Arrow Julia 2.4.0 RC1

+1 (woohoo, first official vote!) On Tue, Oct 25, 2022 at 2:52 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.4.0. > > This release candidate is based on commit: > 571a8fcf6866956d6a47390769e765d1ed0782c7 [1] > > T

Re: [VOTE] Move issue tracking to GitHub Issues

+1 On Wed, Oct 26, 2022 at 5:04 PM Neal Richardson wrote: > I propose that we move issue tracking from the ASF's Jira to GitHub Issues. > This has been discussed on [1] and [2] and there seems to be consensus. A > number of Arrow subprojects already use GitHub Issues; this moves the issue > trac

Re: [VOTE][Julia] Release Apache Arrow Julia 2.4.1 RC0

+1 (tested on macos m1, Julia 1.8.2 and julia#master) On Wed, Nov 16, 2022 at 4:22 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC0) of > Apache Arrow Julia version 2.4.1. > > This release candidate is based on commit: > 23258f12bb4b28eb3846d0d3a91a5

Re: Dictionary Key For Null Slot

I was just looking into a related issue last night where it seems pandas complains if there are _any_ nulls in the dictionary and we were considering not allowing nulls in the dictionary values at all. But it's a little tangled up at the moment because we've already allowed it. Ref: https://github.

Re: [VOTE][Julia] Release Apache Arrow Julia 2.4.2 RC0

+1 (binding) Verified on MacOS m1. -Jacob On Fri, Jan 13, 2023 at 6:17 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC0) of > Apache Arrow Julia version 2.4.2. > > This release candidate is based on commit: > 5ba768918f8088c41e5f89ae890235354a887fd6

Re: [VOTE][Julia] Release Apache Arrow Julia 2.4.3 RC1

+1 Ran on macos m1. -Jacob On Thu, Feb 2, 2023 at 7:53 PM Sutou Kouhei wrote: > +1 > > I ran the following command line on Debian GNU/Linux sid: > > VERIFY_FORCE_USE_JULIA_BINARY=1 dev/release/verify_rc.sh 2.4.3 1 > > > Thanks, > -- > kou > > > In <20230203.113400.196149433832986@cle

Re: [VOTE][Julia] Release Apache Arrow Julia 2.5.0 RC1

+1 Tested on MacOS m1 On Tue, Mar 14, 2023 at 11:56 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.5.0. > > This release candidate is based on commit: > 4d71bee55249dae32983971362256798a9af38bf [1] > > The source

Re: [VOTE][Julia] Release Apache Arrow Julia 2.5.1 RC1

Hmmm, I'm also on MacOS m1, but didn't have any issues running tests. David, is the error reproducible? We fixed an issue for this in [this commit]( https://github.com/apache/arrow-julia/commit/6d0ac4946f062414e2b60aa3d67c2875bb2e7958), but it's possible that our check for this condition wasn't st

Re: [VOTE][Julia] Release Apache Arrow Julia 2.5.1 RC2

Verified on macos m1 +1 -Jacob On Sat, Apr 15, 2023 at 7:19 AM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC2) of > Apache Arrow Julia version 2.5.1. > > This release candidate is based on commit: > e6c44ddbe0fb0c336fad31aa5a84f0b167495d31 [1] > > Th

Re: [VOTE][Julia] Release Apache Arrow Julia 2.5.2 RC1

+1 (macOS M1) -Jacob On Tue, Apr 18, 2023 at 1:59 AM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.5.2. > > This release candidate is based on commit: > 686ab570b831035715cb58f666233ec673e50d8f [1] > > The source re

Re: [VOTE][Julia] Release Apache Arrow Julia 2.6.0 RC1

+1 Tested on apple m1 -Jacob On Sat, Jun 3, 2023 at 3:27 PM Sutou Kouhei wrote: > +1 > > I ran the following command line on Debian GNU/Linux sid: > > VERIFY_FORCE_USE_JULIA_BINARY=1 dev/release/verify_rc.sh 2.6.0 1 > > > Thanks, > -- > kou > > In <20230604.072246.1693870468835902730@cl

Re: [VOTE][Julia] Release Apache Arrow Julia 2.6.1 RC1

+1 (macOS M1) Cheers, -Jacob On Tue, Jun 6, 2023 at 7:48 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.6.1. > > This release candidate is based on commit: > 2d1114e180ef11f9d3bbe310b2eb856550cfbeb3 [1] > > The s

Re: [VOTE][Julia] Release Apache Arrow Julia 2.6.2 RC1

+1 (macOS m1) -Jacob On Fri, Jun 9, 2023 at 1:41 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.6.2. > > This release candidate is based on commit: > 9f1d51a2c975bd83cbaf70c5f640762c6a0bccaf [1] > > The source rel

Re: [ANNOUNCE] New Arrow PMC member: Ben Baumgold,

Yay! Congrats Ben! Love to see more Julia folks here! -Jacob On Tue, Jun 20, 2023 at 4:15 AM Andrew Lamb wrote: > The Project Management Committee (PMC) for Apache Arrow has invited > Ben Baumgold, to become a PMC member and we are pleased to announce > that Ben Baumgold has accepted. > > Congr

Re: [Vote][Format] C Data Interface Format string for REE

+1 (binding) Cheers, -Jacob On Wed, Aug 16, 2023 at 8:16 AM Matt Topol wrote: > Hey All, > > As proposed by Felipe [1] I'm starting a vote on the proposed update to the > Format Spec of adding "+r" as the format string for passing Run-End Encoded > arrays through the Arrow C Data Interface. >

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.0 RC1

+1 Tested on macos m3 with Julia 1.10-rc2 -Jacob On Fri, Dec 8, 2023 at 7:08 PM Dewey Dunnington wrote: > +1 > > I ran > > export PATH="/Applications/ > Julia-1.9.app/Contents/Resources/julia/bin:$PATH" > dev/release/verify_rc.sh 2.7.0 1 > > ...on MacOS M1 Ventura > > On Tue, Dec 5, 2023 at 4:

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.1 RC1

+1, tested on macos. -Jacob On Wed, Jan 31, 2024 at 10:11 AM Ben Baumgold wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.7.1. > > This release candidate is based on commit: > ac199b0e377502ea0f1fa5ced7fda897a01b82a9 [1] > > The s

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.2 RC1

+1, tested on m3 macos On Mon, May 6, 2024 at 4:11 PM Sutou Kouhei wrote: > Hi, > > Note that we already published this version to the official > registry of general Julia packages[1] accidentally[2] but I > would like to start a vote for this version to satisfy the > ASF's release policy[3]. >

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.3 RC1

+1. Tested on macos. On Wed, Aug 28, 2024 at 6:00 PM Sutou Kouhei wrote: > Hi, > > I would like to propose the following release candidate (RC1) of > Apache Arrow Julia version 2.7.3. > > This release candidate is based on commit: > 2696105d01cfda7c55d1902951a20908a3c205e5 [1] > > The source rel

Re: [VOTE] Allow Decimal32 and Decimal64 bitwidths in Arrow Format

Sorry to be late chiming in, but +1 from me. I also have a PR up for the Julia implementation to support the new decimal bitwidths: https://github.com/apache/arrow-julia/pull/525 If there are JSON integration test files that are being updated/included somewhere, I'd love to incorporate them into

RE: Re: sparse data array

> > > On a related note, such encoding would address DataFusion's issue of > > representing scalars / constant arrays: a constant array would be > > represented as a repetition. Currently we just unpack (i.e. allocate) a > > constant array when we want to transfer through a RecordBatch. > In the J

Re: Status of Arrow Julia implementation?

I can comment as the primary apache arrow liaison for the Arrow.jl repository and original code donator. I apologize for the "surprise", but I commented a few times in various places and put a snippet in the README

Re: Status of Arrow Julia implementation?

decided to stop > releasing independently because there wasn't enough development > activity to justify it. > > [1]: https://www.mail-archive.com/dev@arrow.apache.org/msg05971.html > > On Tue, Mar 30, 2021 at 4:54 PM Wes McKinney wrote: > > > > hi Jacob, > &

Re: Status of Arrow Julia implementation?

I hope the development goes well in the future and look forward to > > seeing folks from the Julia ecosystem engaged here on growing the > > Arrow ecosystem. > > > > Thanks, > > Wes > > > > On Fri, Apr 2, 2021 at 3:03 PM Jacob Quinn > wrote: > >

Alignment not stored in arrow metadata

As far as I can tell, the alignment padding used in an IPC stream/file isn't stored explicitly, and not really "inferrable", though maybe technically possible if you calculated what bytes are *necessary* given a buffer's data vs. what's actually stored. Just wondering if this has been brought up a

Re: [DISCUSS] [Rust] Move Rust components to new repos and process

Jorge, * in rust, run integration tests against the latest apache/master on every > PR > I've started to familiarize myself with the archery integration framework over the last few days. Could you clarify for the "archery novices" what exactly ^ this line would mean? Does apache/master refer to t

Re: Status of Arrow Julia implementation?

es for the Julia codebase are in my mind easy to > > > resolve, and if you had indicated that having a customized process for > > > Julia releases was a condition for your joining the community > > > wholeheartedly, we would have been happy to help. I think that the > &

Re: Status of Arrow Julia implementation?

> we'll start voting on Julia package release. > > > Thanks, > -- > kou > > In > "Re: Status of Arrow Julia implementation?" on Sun, 11 Apr 2021 23:06:27 > -0600, > Jacob Quinn wrote: > > > Micah/Wes, > > > > Yes, I've b

Re: Status of Arrow Julia implementation?

; > > > -Micah > > > > On Wed, Apr 14, 2021 at 11:01 PM Sutou Kouhei > wrote: > > > >> Hi Jacob, > >> > >> OK. Here is my plan: > >> > >> 1. We wait for the Rust's move to complete > >> 2. We use a process s

Re: [DISCUSS] Splitting out the Arrow format directory

> > I just thought of one other requirement: the format needs to support > arbitrary byte sequences. > Can you clarify why this is needed? Is it that custom_metadata maps should allow byte sequences as values? On Fri, Aug 13, 2021 at 10:00 AM Phillip Cloud wrote: > On Fri, Aug 13, 2021 at 11:43

Re: [DISCUSS][Julia] How to restart at apache/arrow-julia?

Thanks kou. I think the TODO action list looks good. The one point I think could use some additional discussion is around the release cadence: it IS desirable to be able to release more frequently than the parent repo 3-4 month cadence. But we also haven't had the frequency of commits to necessar

Re: [DISCUSS][Julia] How to restart at apache/arrow-julia?

t; non > >> > trivial and the current 72 hour voting window adds some backpressure > as > >> > well -- I remember Wes may have said windows shorter than 72 hours > might > >> be > >> > fine too) > >> > > >> >

Re: [VOTE] Restart the Julia implementation with new repository and process

+1 On Sun, Sep 26, 2021 at 8:59 PM Sutou Kouhei wrote: > Hi, > > This vote is to determine if the Arrow PMC is in favor of > the Julia community moving the Julia implementation of > Apache Arrow out of apache/arrow into apache/arrow-julia. > > The Julia community uses a process like the Rust com

Re: [RESULT][VOTE] Restart the Julia implementation with new repository and process

Hi kou, I'm looking into the next steps and wondering if it's possible to use the Github mechanism of "transferring a repository" ( https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository), since that could simplify a lot of things. It would retain exis

Re: [RESULT][VOTE] Restart the Julia implementation with new repository and process

IP clearance. Some items listed in > > https://incubator.apache.org/ip-clearance/ does so such as > > https://incubator.apache.org/ip-clearance/daffodil-vscode-debugger > > . > > > > > > Thanks, > > -- > > kou > > > > In > > "Re

Re: [RESULT][VOTE] Restart the Julia implementation with new repository and process

dual CLA. > > I think that people who commit after > https://incubator.apache.org/ip-clearance/arrow-julia-library.html > (8583da8a84a9e355affb42654dcd8c765bcc3134) are target: > > $ git shortlog -sn > 8583da8a84a9e355affb42654dcd8c765bcc3134..1447cb2b13b728729f9a89760ac07a848e

Julia implementation and integration with main apache arrow repository

Hello all, Hopefully this email works (I'm not super familiar with using mailing lists like this). Over the past few weeks, I've been working on a pure Julia implementation to support serializing/deserializing the arrow format for Julia. The code in its current state can be found here: https://gi

Compression?

Am I correct in understanding there's nothing in the arrow ipc/file format spec about compression? I thought I had seen something at one point, but looking over the spec website, I don't see anything. -Jacob

Re: Compression?

remains on the > roadmap for the project. > > [1]: https://github.com/apache/arrow/blob/master/format/Message.fbs#L45 > > On Tue, Sep 15, 2020 at 7:18 PM Jacob Quinn > wrote: > > > > Am I correct in understanding there's nothing in the arrow ipc/file > forma

Re: [DISCUSS] Rethinking our approach to scheduling CPU and IO work in C++?

My immediate thought reading the discussion points was Julia's task-based multithreading model that has been part of the language for over a year now. An announcement blogpost for Julia 1.3 laid out some of the details and high-level approach: https://julialang.org/blog/2019/07/multithreading/, and

Re: Dictionary key access in python/generally

> > But I'm also attaching table > metadata to each feather, which I'd hate to lose. > Note the arrow format allows attaching custom metadata at the column (field), schema, and message level, so it should be possible to retain any metadata this way. -Jacob On Wed, Oct 7, 2020 at 11:38 AM Benjami

Re: Github Actions feedback time

>From this page, it looks like there have been certain github organizations that have been "whitelisted" to allow their github actions to run. Is there a process to do this whitelisting? If the `julia-actions` github org was allowed to run, that would enable everything needed for Julia CI to run.

Re: Julia package

Hi Krisztián, I explained a little bit the setup [here]( https://github.com/apache/arrow/pull/9121#discussion_r554149673) recently. We're still in a transition from the JuliaData repo to apache/arrow in terms of development (traditionally Julia packages are their own github repos, so users have be

Re: [VOTE] Release Apache Arrow 3.0.0 - RC0

I found a small issue with the Julia installation instructions; a PR to fix is here: https://github.com/apache/arrow/pull/9226. With that change, the Julia package can be installed and tests pass for me locally. -Jacob On Fri, Jan 15, 2021 at 3:53 PM Krisztián Szűcs wrote: > Hi, > > I would lik

Re: [ANNOUNCE] Apache Arrow 3.0.0 released

Can we make sure Julia gets added to the language list in the future? ;) On Tue, Jan 26, 2021 at 6:45 AM Krisztián Szűcs wrote: > The Apache Arrow community is pleased to announce the 3.0.0 release. > The release includes 678 resolved issues ([1]) since the 2.0.0 release. > > The release is avai

Re: Constraints on fixed size list of variables sized types

Yeah, I didn't quite follow the example either; it seems like your example actually corresponds to a FixedSizeList[2]>[2]? Or perhaps FixedSizeList>[2]? Assuming the former, it seems you'd need additional fixed size slots to account for the Null element. In Julia, you can inspect the internal struc

No replacement dictionaries supported in pyarrow?

Had an issue come up here: https://github.com/JuliaData/Arrow.jl/issues/129#issuecomment-777350450. >From the implementation status page, it says C++ supports replacement dictionaries and that python tracks the C++ implementation. Is this just a pyarrow issue where it specifically doesn't support r

Re: No replacement dictionaries supported in pyarrow?

lowed > > by > > > one or more dictionary/record-batch messages). > > > > > > -Micah > > > > > > [1] https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format > > > > > > On Wed, Mar 17, 2021 at 11:04 PM Jacob Quinn

Question on Exactness of Arrow Memory Layout

Having become familiar with the Arrow memory layout, and taking a stab at an implementation in the Julia language, I've come up with a perhaps naive question. A "type" (class) I have defined so far is: immutable Column{T} <: ArrowColumn{T} buffer::Vector{UInt8} # potential reference to mmap

Re: [VOTE][Julia] Release Apache Arrow Julia 2.7.4 RC1

+1 Tested on macos m3. -Jacob On Mon, Oct 28, 2024 at 2:48 PM Sutou Kouhei wrote: > +1 (binding) > > I ran the following command line on Debian GNU/Linux sid: > > VERIFY_FORCE_USE_JULIA_BINARY=1 dev/release/verify_rc.sh 2.7.4 1 > > > Thanks, > -- > kou > > In <20241029.053820.117383943380586

Re: [RESULT][VOTE][Julia] Release Apache Arrow Julia 2.7.4 RC1

I believe it's only 3 +1, but that's still the required amount for the vote to carry. -Jacob On Tue, Oct 29, 2024 at 6:15 PM Sutou Kouhei wrote: > Hi, > > The vote carries with: > > 4 +1 (binding): Sutou Kouhei, Jacob Quinn, Dewey Dunnington > 0 0: > 0 -1: &

Re: [VOTE][Julia] Release Apache Arrow Julia 2.8.0 RC1

-1. I discovered an info log statement that should be a debug statement. Doesn't affect tests passing, but introduces a little spam printing. Proposed a PR to fix and it was merged here: https://github.com/apache/arrow-julia/pull/537/files. -Jacob On Tue, Dec 3, 2024 at 12:21 AM Sutou Kouhei wr

Re: [VOTE][Julia] Release Apache Arrow Julia 2.8.0 RC2

+1 Verified on macos m3 -Jacob On Thu, Dec 5, 2024 at 2:46 AM Raúl Cumplido wrote: > +1 (binding) > > Verified on Ubuntu 24.04 > > El jue, 5 dic 2024 a las 2:26, David Li () escribió: > > > +1 (binding) > > > > Tested on Debian 12/x86_64 > > > > On Thu, Dec 5, 2024, at 10:05, Sutou Kouhei wro

Re: [VOTE] Enable GitHub Discussions for apache/arrow-*

+1 On Thu, Mar 20, 2025 at 10:50 PM Sutou Kouhei wrote: > Hi, > > I would like to propose enabling GitHub Discussions on: > > * apache/arrow > * apache/arrow-adbc > * apache/arrow-cookbook > * apache/arrow-experiments > * apache/arrow-flight-sql-postgresql > * apache/arrow-go > * apache/arrow-ja

Fwd: How to concatenate RecordBatches into a single RecordBatch?

Hi all, Question: If I have a set of small (10-1000 rows) RecordBatches on disk or in memory, how can I (efficiently) concatenate/rechunk them into larger RecordBatches (so that each column is output as a contiguous array when written to a new Arrow buffer)? Context: With such small RecordBatches