Re: Github Actions feedback time

Hi Wes, I fielded ARROW-11140 with what I would consider a successful POC from Rust implementation's point of view. Let's move the discussion there. I was already able to run Rust's first steps there (using my own mac as a runner), and, so far, l

Re: Arrow 3.0 release

Hi Andrew, We need to fix nightly builds to release a new version. Could you check the latest nightly builds[1], open JIRA issues for not unaddressed failed builds and fix them one by one? [1] https://lists.apache.org/thread.html/ra17f2f9c2323498edd45665d78ce669ae90c0bbd888921f94cf9ca5c%40%3Cde

Re: Arrow 3.0 release

Hi, > I personally haven't been able to get a build working. Could you open a JIRA issue and share the details such as how to reproduce, full build log and so on? Thanks, -- kou In "Re: Arrow 3.0 release" on Mon, 4 Jan 2021 16:44:23 -0800, "M. Edward (Ed) Borasky" wrote: > P.S.: I can t

Re: Github Actions feedback time

Random question - could Github be reached out to for the limits to be increased/investigated in the mean time, given this is an opensource project? On Wed, 6 Jan 2021 at 06:57, Andrew Lamb wrote: > Makes sense -- thank you > > On Tue, Jan 5, 2021 at 5:38 PM Wes McKinney wrote: > > > There are a

Re: Arrow January board report -- help needed

I can look into stats for project activity and community health. It's not totally clear to me what kind of Issues should be shared in this context, but I have two on my mind: * The way INFRA crippled the CI for the most active parts of our project (Rust, C++) without warning on a holiday weekend,

Re: Github Actions feedback time

Makes sense -- thank you On Tue, Jan 5, 2021 at 5:38 PM Wes McKinney wrote: > There are a lot of companies now with a business interest in the success of > Apache Arrow. We should do a back of the envelope estimate of the number of > compute hours we use per year and arrive at an annual cost est

Re: Github Actions feedback time

There are a lot of companies now with a business interest in the success of Apache Arrow. We should do a back of the envelope estimate of the number of compute hours we use per year and arrive at an annual cost estimate. My guess (and maybe I’m wildly off) is that it’s somewhere around $50,000-100,

Re: Github Actions feedback time

One thing that is not clear to me is who will pay for the dedicated infrastructure costs with something like Build Kite? I don't fully understand how moving to self hosted (and thus also presumably limited) arrow-only infrastructure will reduce build queue times, unless there is a deep pocketed sp

Re: Github Actions feedback time

hi Jorge -- as far as I know, all that's necessary to use BK is the pipeline configuration in the repo and to attach workers to Arrow's (free) account (which I'm happy to add you to). We can likely get a lot of free cloud credits from Google, Microsoft, or Amazon if we wanted to go that route to ke

Arrow January board report -- help needed

It's time for our quarterly board report (due next Wednesday). Can everyone help with completing the various sections? ## Description: The mission of Apache Arrow is the creation and maintenance of software related to columnar in-memory processing and data interchange ## Issues: [Insert your own

Re: Github Actions feedback time

Krisztián, I agree with you that there is an ongoing problem with the queue. Thanks for the tips wrt to WIP. Wes, I would be up for moving Rust workflows to Buildkite. Is the integration in place wrt to reporting to triggers and reporting back to gihtub? I.e. can we just place a `pipeline.yml` on

Arrow sync call January 6 at 12:00 US/Eastern, 17:00 UTC

Hi all, Reminder that our biweekly call is in 24 hours at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be shared with the mailing list afterward. Neal

Re: Github Actions feedback time

At the risk of sounding like a broken record -- we are almost certainly going to have to move our builds to dedicated infrastructure that this community has complete agency over sometime between now and 2025. Maybe it will be this year, maybe next year, but to me it is an inevitability. I spent qui

Re: Github Actions feedback time

On Tue, Jan 5, 2021 at 4:40 PM Jeroen Ooms wrote: > > On Tue, Jan 5, 2021 at 1:34 PM Krisztián Szűcs > wrote: > > > > Hi, > > > > I'm concerned about the overall feedback time we have on pull requests. > > I have a simple PR to make the comment bot working again, but no > > builds are running eve

Re: Github Actions feedback time

On Tue, Jan 5, 2021 at 1:34 PM Krisztián Szűcs wrote: > > Hi, > > I'm concerned about the overall feedback time we have on pull requests. > I have a simple PR to make the comment bot working again, but no > builds are running even after 30 minutes. FWIW, GitHub seems to queue jobs by pusher, not

Re: Github Actions feedback time

On Tue, Jan 5, 2021 at 3:21 PM Jorge Cardoso Leitão wrote: > > Hi Krisztian, > > Could you describe which flows you are referring to? > > Rust builds are taking 4-6m on non-(windows/mac) and 8-15m on windows/mac, > with almost immediate feedback. E.g. I force-pushed recently > https://github.com/a

Re: Github Actions feedback time

Hi Krisztian, Could you describe which flows you are referring to? Rust builds are taking 4-6m on non-(windows/mac) and 8-15m on windows/mac, with almost immediate feedback. E.g. I force-pushed recently https://github.com/apache/arrow/pull/9089 and the first complete test came 5m later. One hypo

Github Actions feedback time

Hi, I'm concerned about the overall feedback time we have on pull requests. I have a simple PR to make the comment bot working again, but no builds are running even after 30 minutes. I can see 2-4 running builds, which will make our work much harder right before the release. I wasn't following th

Re: Arrow 3.0 release

On Tue, Jan 5, 2021 at 1:46 AM Andrew Wieteska wrote: > > Dear all, > > I will have time and would be happy to help with the release if the PMC > release manager could use a second person. I don't have experience with the > release process so I might need some detailed pointers on where and how I

[NIGHTLY] Arrow Build Report for Job nightly-2021-01-05-0

Arrow Build Report for Job nightly-2021-01-05-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2021-01-05-0 Failed Tasks: - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2021-01-05-0-github-test-conda-pyt