[C++] Random table generator and table converter

Hi, For the C++ tests for the ORC writer there are two functions I need which can significantly shorten the tests, namely a generic table generator and a table converter. For the former I know there is arrow/testing/random.h which can generate random arrays. Shall I generate random struct arr

Re: [RESULT] [VOTE] Release Apache Arrow 3.0.0 - RC2

1. [done] rebase master 2. [done] upload source 3. [done] upload binaries 4. [done] update website 5. [done] upload ruby gems 6. [done] upload js packages 8. [done] upload C# packages 9. [done] upload rust crates 10. [done] update conda recipes 11. [done] upload wheels/sdist to pypi 12. [do

Re: Pandas Block Manager

Hi all - just pinging this thread given the later discussions on the PR . I am proposing a backwards (but not forwards) compatible change to the spec to strike this line out When serializing Arrow data for interprocess communication, these alignment and pa

Re: ursa-labs/crossbow on travis-ci.com is disabled

Hi, We released 3.0.0. We can stop the Travis CI subscription. Thanks! We should prepare new (Buildkite based?) CI for arm64 by the next release. Thanks, -- kou In <20210112.040015.1663654805299715665@clear-code.com> "Re: ursa-labs/crossbow on travis-ci.com is disabled" on Tue, 12 Jan 20

Re: [ANNOUNCE] Apache Arrow 3.0.0 released

Can we make sure Julia gets added to the language list in the future? ;) On Tue, Jan 26, 2021 at 6:45 AM Krisztián Szűcs wrote: > The Apache Arrow community is pleased to announce the 3.0.0 release. > The release includes 678 resolved issues ([1]) since the 2.0.0 release. > > The release is avai

Re: Arrow Rust Sync Call 1/27/2021

I think another thing that got mentioned on the call that might be worth adding to the agenda for the next meeting was something Jorge said along the lines of: "The original Rust Arrow implementation was modeled after the C++ implementation. As it matures, it seems like we are heading more towards

Re: [RUST] Implement value function with Array trait

I think the idea is enticing, but it comes with some challenges: 1. Rust is strongly typed so when extracting values we would likely need a `Scalar` type enum or multiple different `value_bool`, `value_u64` type functions 2. Such access would likely be much slower (though possible more convenient)

Re: [RESULT] [VOTE] Release Apache Arrow 3.0.0 - RC2

Thank you -- avoiding force pushes to master will make outstanding PRs simpler to understand and it easier to consume arrow by downstream projects Andrew On Wed, Jan 27, 2021 at 8:49 AM Wes McKinney wrote: > Right, that’s what we discussed before this release, so I want to confirm > that we are

[Rust] Upgrade to tokio 1.0.0 -- RFC

I would like to bring your awareness to a PR that proposes to upgrade datafusion / arrow flight to use tokio 1.0: https://github.com/apache/arrow/pull/9230 Given that tokio 1.0 is not compatible with versions prior to 1.0 this upgrade will mean, effectively, that all projects that depend on arrow

Re: Introducing Buzz, Arrow powered serverless query engine

This is really interesting Rémi! I like the interesting take on using "serverless" cloud components to build a scalable query engine. Looking forward to learn more about it! Op wo 27 jan. 2021 om 17:58 schreef Andy Grove : > Thanks for sharing this, Rémi! It looks really interesting and it is gr

Re: Arrow Rust Sync Call 1/27/2021

Looks like I put this on my calendar wrong, but really glad this is happening, will be there next call! We are doing a lot in rust and arrow this year and super excited you got this started Andy! On Wed, Jan 27, 2021 at 9:58 AM Rémi Dettai wrote: > thanks Andy!! > > Le mer. 27 janv. 2021 à 18:4

Re: Arrow Rust Sync Call 1/27/2021

thanks Andy!! Le mer. 27 janv. 2021 à 18:41, Andy Grove a écrit : > Attendees > >- > >Mahmut Bulut >- > >Remi Dettai >- > >Andy Grove >- > >Fernando Herrera >- > >Jorn Horstmann >- > >Andrew Lamb >- > >Jorge Leitao >- > >Mike Seddon

Arrow Rust Sync Call 1/27/2021

Attendees - Mahmut Bulut - Remi Dettai - Andy Grove - Fernando Herrera - Jorn Horstmann - Andrew Lamb - Jorge Leitao - Mike Seddon Topics Discussed - Release process - There is an email thread "[Proposal] Modify release pro

Re: Introducing Buzz, Arrow powered serverless query engine

Thanks for sharing this, Rémi! It looks really interesting and it is great to see another project powered by the Rust implementation of Arrow. On Tue, Jan 26, 2021 at 10:18 AM Rémi Dettai wrote: > Hi all, > > I have been following this community for nearly a year now, trying to > contribute when

Re: [Rust] Proposed discussion items for the Rust sync up meeting this Wednesday Jan 27, 2021

Can't be there at the Rust sync call. Hope to be able to join next time. I will contribute some bits for the blog post. Would be nice if there would be some notes of the sync call (very high level is fine). Daniël Op wo 27 jan. 2021 om 16:16 schreef Andy Grove : > I had an action from the last

Re: [Rust] Proposed discussion items for the Rust sync up meeting this Wednesday Jan 27, 2021

I had an action from the last call to create a Google doc for us to collaborate on for a blog post announcing 3.0.0. Here is the document. I didn't get a chance to actually fill out much but hopefully, we can all contribute to this. https://docs.google.com/document/d/1HfdmVE9iQcYlWyfk68BN2PcIjAAhR

Re: Should we default to write parquet format version 2.0? (not data page version 2.0)

Coming back to this topic, there is one additional aspect that might warrant some consideration: forward compatibility for nanosecond timestamps. If we would switch to a default of `version="2.0"`, that would mean we start using the LogicalType with nanosecond time unit. Since this has no equivale

Re: [RESULT] [VOTE] Release Apache Arrow 3.0.0 - RC2

Right, that’s what we discussed before this release, so I want to confirm that we aren’t going to rebase master anymore. I’ll bring it up the next release to make sure we don’t do it again. On Wed, Jan 27, 2021 at 7:45 AM Krisztián Szűcs wrote: > On Wed, Jan 27, 2021 at 2:35 PM Wes McKinney wro

Re: [RESULT] [VOTE] Release Apache Arrow 3.0.0 - RC2

On Wed, Jan 27, 2021 at 2:35 PM Wes McKinney wrote: > > I want to confirm that this is the last time that we are going to rebase > master? In order to avoid that we need to release from a maintenance branch, which should be possible with minor modifications. > > On Wed, Jan 27, 2021, 3:00 AM Uwe L

Re: [RESULT] [VOTE] Release Apache Arrow 3.0.0 - RC2

I want to confirm that this is the last time that we are going to rebase master? On Wed, Jan 27, 2021, 3:00 AM Uwe L. Korn wrote: > 1. [done] rebase master > 2. [done] upload source > 3. [done] upload binaries > 4. [done] update website > 5. [done] upload ruby gems > 6. [done] upload js pa

Re: [Rust] Proposed discussion items for the Rust sync up meeting this Wednesday Jan 27, 2021

Hi, Thanks a lot for driving this, Andrew. I agree with what both of you wrote here :) Best, Jorge On Tue, Jan 26, 2021 at 6:06 PM Rémi Dettai wrote: > Great topics Andrew, to my knowledge nothing has been decided on these > topics. > > We also agreed last time that it would be nice to go ro

[RUST] Implement value function with Array trait

Hi, I'm wondering if it has been considered to move the value function that is implemented in all the arrays (StringArray, BooleanArray, ListArray, etc) as part of the Array trait? This would help when extracting values from generic arrays that implement dyn Array without having to manually downc

[NIGHTLY] Arrow Build Report for Job nightly-2021-01-27-0

Arrow Build Report for Job nightly-2021-01-27-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-01-27-0 Failed Tasks: - centos-8-aarch64: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-01-27-0-travis-centos-8-aarch64 - con

Re: [RESULT] [VOTE] Release Apache Arrow 3.0.0 - RC2

1. [done] rebase master 2. [done] upload source 3. [done] upload binaries 4. [done] update website 5. [done] upload ruby gems 6. [done] upload js packages 8. [done] upload C# packages 9. [done] upload rust crates 10. [done] update conda recipes 11. [done] upload wheels/sdist to pypi 12. [ ]