Re: CI feedback time

2021-04-15 Thread Jorge Cardoso Leitão
Hi, I agree. I'll submit two requirements though: > - the configuration for CI builds must be kept in the Arrow repository >(as they are currently in .github, etc.) > - CI builds must be runnable from PRs > I'll submit three more: - The result of the build (pass / did not pass) must be shown

[VOTE] Release Apache Arrow 4.0.0 - RC0

2021-04-15 Thread Krisztián Szűcs
Hi, I would like to propose the following release candidate (RC0) of Apache Arrow version 4.0.0. This is a release consisting of 671 resolved JIRA issues[1]. This release candidate is based on commit: 3df78d3a98f346ed09667edc5ab551cfeff50b7a [2] The source release rc0 is hosted at [3]. The binar

Re: CI feedback time

2021-04-15 Thread Krisztián Szűcs
On Fri, Apr 16, 2021 at 1:11 AM Jed Brown wrote: > > Wes McKinney writes: > > > I think we should take a more serious look at Buildkite for some of our CI. > > > > * First of all, it's very easy to connect self-hosted workers and > > supports ephemeral cloud workers in a way that would be difficu

Re: CI feedback time

2021-04-15 Thread Jed Brown
Wes McKinney writes: > I think we should take a more serious look at Buildkite for some of our CI. > > * First of all, it's very easy to connect self-hosted workers and > supports ephemeral cloud workers in a way that would be difficult or > impossible with GHA. No need to have Infra fiddle with

Re: CI feedback time

2021-04-15 Thread Krisztián Szűcs
On Thu, Apr 15, 2021 at 11:53 PM Andy Grove wrote: > > I started looking at BulidKite and it would solve one large problem for the > DataFusion/Ballista project. We really need to be running integration tests > against large data sets (such as TPC-H @ SF=100GB) and self-hosted > BuildKite makes th

Re: CI feedback time

2021-04-15 Thread Andy Grove
I started looking at BulidKite and it would solve one large problem for the DataFusion/Ballista project. We really need to be running integration tests against large data sets (such as TPC-H @ SF=100GB) and self-hosted BuildKite makes this simple to accomplish. I even have some modest hardware that

Re: CI feedback time

2021-04-15 Thread Wes McKinney
I think we should take a more serious look at Buildkite for some of our CI. * First of all, it's very easy to connect self-hosted workers and supports ephemeral cloud workers in a way that would be difficult or impossible with GHA. No need to have Infra fiddle with the admin dashboard. So we could

Re: [Java] Source control of generated flatbuffers code

2021-04-15 Thread Bob Tinsman
OK, I just approved those changes. I was working on a shell script to automate it--nice to have, but not necessary. Better that you can get it into 4.0. Thanks! On 2021/04/15 17:33:20, Micah Kornfield wrote: > I took a look and added comments. I'm not sure if Bob replied off-list, > so hopef

[Rust][Datafusion] Timestamp Millisecond support

2021-04-15 Thread Evan Chan
Hi folks, So currently Arrow Rust/DataFusion supports four types of Timestamp arrays, with Nano, Micro, Millisecond and Second resolution. However, the best supported by far are Nanos. For example, in DataFusion, the following only works for Nanos and not the other resolutions: * CAST(x as TI

Re: [Java] Source control of generated flatbuffers code

2021-04-15 Thread Micah Kornfield
I took a look and added comments. I'm not sure if Bob replied off-list, so hopefully no work was duplicated. Lets try to be mindful that the project is asynchronous in nature and it might take a little time to reply. Cheers, Micah On Thu, Apr 15, 2021 at 10:00 AM Nate Bauernfeind < natebauernf

Re: [Java] Source control of generated flatbuffers code

2021-04-15 Thread Nate Bauernfeind
> I think checking in the java files is fine and probably better then relying > on a third party package. We should make sure there are instructions on > how to regenerate them along with the PR Micah, I just opened a pull-request to satisfy ARROW-12111. This is my first contribution to an apach

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread paddy horan
+1 From: Joris Van den Bossche Sent: Thursday, April 15, 2021 10:07:27 AM To: dev Subject: Re: [VOTE] Move Rust components to new repos and process +1 (non-binding) Joris On Thu, 15 Apr 2021 at 15:42, Wes McKinney wrote: > +1 (binding) > > On Thu, Apr 15, 2

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread Joris Van den Bossche
+1 (non-binding) Joris On Thu, 15 Apr 2021 at 15:42, Wes McKinney wrote: > +1 (binding) > > On Thu, Apr 15, 2021 at 7:31 AM Weston Steimel > wrote: > > > > +1 > > > > On Thu, 15 Apr 2021 at 00:05, Andy Grove wrote: > > > > > This vote is to determine if the Arrow PMC is in favor of the Rust >

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread Wes McKinney
+1 (binding) On Thu, Apr 15, 2021 at 7:31 AM Weston Steimel wrote: > > +1 > > On Thu, 15 Apr 2021 at 00:05, Andy Grove wrote: > > > This vote is to determine if the Arrow PMC is in favor of the Rust > > community moving the Rust implementation of Apache Arrow as well as the > > related projects

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread Weston Steimel
+1 On Thu, 15 Apr 2021 at 00:05, Andy Grove wrote: > This vote is to determine if the Arrow PMC is in favor of the Rust > community moving the Rust implementation of Apache Arrow as well as the > related projects (such as Parquet, DataFusion, Ballista, etc) out of the > monorepo and into two new

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread Antoine Pitrou
+0. Regards Antoine. Le 15/04/2021 à 02:04, Andy Grove a écrit : This vote is to determine if the Arrow PMC is in favor of the Rust community moving the Rust implementation of Apache Arrow as well as the related projects (such as Parquet, DataFusion, Ballista, etc) out of the monorepo and i

Re: CI feedback time

2021-04-15 Thread Krisztián Szűcs
On Thu, Apr 15, 2021 at 2:13 AM Weston Pace wrote: > > It may be worth reaching out to the Airflow project. Based on > https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status > it seems they have been investing time into figuring how to make > self-hosted runners work (it seems G

Re: CI feedback time

2021-04-15 Thread Krisztián Szűcs
On Thu, Apr 15, 2021 at 10:48 AM Antoine Pitrou wrote: > > > Le 15/04/2021 à 03:13, Kazuaki Ishizaki a écrit : > > As we know this is a common issue among Apache projects. While the > > projects do not have the final solution, Apache Spark project has a > > mechanism [1][2] to run a test in own lo

Re: [Rust] [DataFusion] Proposal for datafusion test reorganization

2021-04-15 Thread Andrew Lamb
Thanks Daniël, I'll write up a more formal proposal / jira in the upcoming days Andrew On Tue, Apr 13, 2021 at 11:37 AM Daniël Heres wrote: > Late reply, but I agree these tests modules need a bit of reorganization. I > also found myself adding tests to context.rs / sql.rs just because > relat

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread Daniël Heres
+1 Op do 15 apr. 2021 om 12:37 schreef Andrew Lamb : > +1 > > On Thu, Apr 15, 2021 at 1:17 AM Fernando Herrera < > fernando.j.herr...@gmail.com> wrote: > > > +1 > > > > On Thu, 15 Apr 2021, 05:57 Sutou Kouhei, wrote: > > > > > +1 > > > > > > In > > > > "[VOTE] Move Rust components to new repo

Re: [VOTE] Move Rust components to new repos and process

2021-04-15 Thread Andrew Lamb
+1 On Thu, Apr 15, 2021 at 1:17 AM Fernando Herrera < fernando.j.herr...@gmail.com> wrote: > +1 > > On Thu, 15 Apr 2021, 05:57 Sutou Kouhei, wrote: > > > +1 > > > > In > > "[VOTE] Move Rust components to new repos and process" on Wed, 14 Apr > > 2021 18:04:44 -0600, > > Andy Grove wrote: >

[NIGHTLY] Arrow Build Report for Job nightly-2021-04-15-0

2021-04-15 Thread Crossbow
Arrow Build Report for Job nightly-2021-04-15-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-15-0 Failed Tasks: - conda-linux-gcc-py36-arm64: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-15-0-drone-conda-linux-g

Re: CI feedback time

2021-04-15 Thread Antoine Pitrou
Le 15/04/2021 à 03:13, Kazuaki Ishizaki a écrit : As we know this is a common issue among Apache projects. While the projects do not have the final solution, Apache Spark project has a mechanism [1][2] to run a test in own local (forked) repository. Can we alleviate the problem a little bit?