Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-06 Thread Hyukjin Kwon
Hi all, I am an Apache Spark PMC, and would like to know the future plan about GitHub Actions in ASF. Please also see the INFRA ticket I filed: https://issues.apache.org/jira/browse/INFRA-21646. I am aware of the limited GitHub Actions resources that are shared across all projects in ASF, and man

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-07 Thread Hyukjin Kwon
suffer from the lack of resources. I appreciate the resources provided to us but that does not resolve the issue of the development being slowed down. 2021년 4월 7일 (수) 오후 5:52, Greg Stein 님이 작성: > On Wed, Apr 7, 2021 at 12:25 AM Hyukjin Kwon wrote: > >> Hi all, >> >>

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-07 Thread Hyukjin Kwon
anisation shares all resources across the projects. 2021년 4월 7일 (수) 오후 10:04, Martin Grigorov 님이 작성: > > > On Wed, Apr 7, 2021 at 3:41 PM Hyukjin Kwon wrote: > >> Hi Greg, >> >> I raised this thread to figure out a way that we can work together to >> resolve this

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-07 Thread Hyukjin Kwon
incentive to optimize > their own usage. I imagine this would be the best thing since the sliced > bread that INFRA could provide to all the projects. Thanks Jarek. I think this sounds reasonable and realistic to me. +1 2021년 4월 7일 (수) 오후 10:30, Hyukjin Kwon 님이 작성: > Thanks Martin fo

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-12 Thread Hyukjin Kwon
een our > infra/CI.yml) > - manageability (assigning resources per-project) - this works by having > self-hosted runners assigned per project (we needed infra JIRA ticket and > generation of a bunch of tokens for our runners and our own AWS account > with auto-scaling). > > It would b

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-15 Thread Hyukjin Kwon
Hi all, Is here the right place to expect feedback from the infra team or related people? It would be great to hear what the infra team thinks about Jarek's suggestion. 2021년 4월 13일 (화) 오전 11:15, Hyukjin Kwon 님이 작성: > Hi all, > > Could we have any update and feedback from the IN

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-16 Thread Hyukjin Kwon
asonable and realistic to me. How do you think about this? On Fri, 16 Apr 2021, 21:36 Martin Grigorov, wrote: > Hi Hyukjin, > > On Fri, Apr 16, 2021 at 3:04 AM Hyukjin Kwon wrote: > > > Hi all, > > > > Is here the right place to expect feedback from the infra team

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-16 Thread Hyukjin Kwon
what has changed if I am supposed to independently discuss with the GitHub team to address this now. On Fri, 16 Apr 2021, 22:00 Martin Grigorov, wrote: > > > On Fri, Apr 16, 2021 at 3:44 PM Hyukjin Kwon wrote: > >> I thought Jarek was pretty clear on that. I meant this: >&g

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-16 Thread Hyukjin Kwon
quite yet! > > On Fri, Apr 16, 2021 at 07:44 Hyukjin Kwon wrote: > > > I thought Jarek was pretty clear on that. I meant this: > > > > > So it all has to start with 'per-project' resource limitation and self- > > > budgeting. It would be GREAT if infra.c

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-16 Thread Hyukjin Kwon
; > I emailed Github a few days ago asking a bunch of questions; > and hope to get another builds@ meeting with Github as guests pretty soon. > > Will let you know how it goes > > > > On Fri, Apr 16, 2021 at 3:46 PM Hyukjin Kwon wrote: > > > Yes, per project b

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-19 Thread Hyukjin Kwon
Thanks all. Just to add a bit of note, > * Create a wiki page collecting all the practices to reduce the hours > (using the pr cancel workflow discussed earlier + timeouts + ...?) We should probably also mention that Apache Spark managed to distribute the workflow runs to forked repositories in

Re: Github hard limit on job execution time

2023-04-13 Thread Hyukjin Kwon
Just dropping a comment. Apache Spark solved it by splitting the job. As of the number of parallel jobs, Apache Spark made, in PR builder, a custom logic to link the GitHub workflow run in forked repositories - so we reuse the GitHub resources in PR authors forked repository instead of the one all