Yes, per project budget will be great. Beside cancellation workaround, Apache Spark is trying another workaround for the time being: distributing workflow runs to forked repositories to leverage contributor's GitHub Actions resources instead of consuming all ASF organisation resources.
Please see also how I implemented it if anyone is interested in this - the PR description should be self-explanatory: - https://github.com/apache/spark/pull/32092 - https://github.com/apache/spark/pull/32193 Note that this is still a workaround and it disables GitHub Actions work out of the box. On Fri, 16 Apr 2021, 22:26 Matt Sicker, <boa...@gmail.com> wrote: > I thought one of the key points raised by Jarek before was that even with > infinite compute resources, unoptimized actions will fill up all the > compute (like how widening highways induces higher traffic demand). The > per-PMC compute budgets sounds like a great way to help “shift left” on the > problem to avoid demanding Infra fix hundreds of scripts they know nothing > about. The other optimizations Jarek has demonstrated here like ensuring > old commits get cancelled in favor of new commits, or more sophisticated > things like only running tests that are relevant to the commit, can all go > a long way toward optimizing build compute. > > Now if you’re attempting to run entire integration and end to end test > suites on every commit, I don’t think the earth has enough compute power > for that quite yet! > > On Fri, Apr 16, 2021 at 07:44 Hyukjin Kwon <gurwls...@gmail.com> wrote: > > > I thought Jarek was pretty clear on that. I meant this: > > > > > So it all has to start with 'per-project' resource limitation and self- > > > budgeting. It would be GREAT if infra.could provide self-hosted GitHub > > > Runners SERVICE per project, where project could donate credits or > money > > > for their own account, then the projects would have incentive to > optimize > > > their own usage. I imagine this would be the best thing since the > sliced > > > bread that INFRA could provide to all the projects. > > > > Maintaining and providing a self-hosted runners in GitHub Actions where > the > > resources are managed in project level where each project can donate > > credits. > > > > In addition, Jarek mentioned that Airflow already has a working version - > > is it correct Jarek? > > > > If the infra team takes and improves it for other ASF projects, that > would > > permanently resolve this issue. > > > > This suggestion looks reasonable and realistic to me. > > > > How do you think about this? > > > > > > On Fri, 16 Apr 2021, 21:36 Martin Grigorov, <mgrigo...@apache.org> > wrote: > > > > > Hi Hyukjin, > > > > > > On Fri, Apr 16, 2021 at 3:04 AM Hyukjin Kwon <gurwls...@gmail.com> > > wrote: > > > > > > > Hi all, > > > > > > > > Is here the right place to expect feedback from the infra team or > > related > > > > people? > > > > It would be great to hear what the infra team thinks about Jarek's > > > > suggestion. > > > > > > > > > > What suggestion exactly do you mean ? > > > I've just re-read Jarek's email and I see 3 tasks for Github Actions > > team, > > > but nothing specific for Apache Infra team. > > > > > > > > > > > > > > > > > > 2021년 4월 13일 (화) 오전 11:15, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > > > > > > > >> Hi all, > > > >> > > > >> Could we have any update and feedback from the INFRA team about > > Jarek's > > > >> suggestion please? > > > >> > > > >> 2021년 4월 9일 (금) 오전 7:06, Jarek Potiuk <ja...@potiuk.com>님이 작성: > > > >> > > > >>> > > > >>>> That's a good idea. We do need to thank Github to give free > > resources > > > to > > > >>>> ASF projects, but it's better if we can make it a business: we > allow > > > >>>> individual projects to sign deals with Github to get dedicated > > > >>>> resources. > > > >>>> It's a bit wasteful to ask every project to set up its own dev > ops, > > > >>>> using Github Action is more convenient. Maybe we should raise it > to > > > >>>> Github? > > > >>>> > > > >>> > > > >>> I do not think you can get per-project resources in GH - the most > you > > > >>> can do are self-hosted runners for your project. > > > >>> > > > >>> (BTW I am not from the INFRA team - just a humble "CI person" of > > Apache > > > >>> Airflow but very much vested into Github Actions) > > > >>> maybe the infra team can chime in here. We did raise it to GitHub, > we > > > >>> even had meeting with them > > > >>> organized by Gavin and several topics were raised that could be > > > >>> eventually addressed by Github: > > > >>> > > > >>> - observability (they could not give us per-project usage > dashboard - > > > we > > > >>> built our own imperfect (with API limitations) one by Tobiasz from > > > Airllow > > > >>> - security (limiting access to only project committers) - this we > > > >>> handled by the Ash's fork of Runner (but it's also imperfect - even > > > today I > > > >>> had to fix a problem where we had list of committers desynchronised > > > between > > > >>> our infra/CI.yml) > > > >>> - manageability (assigning resources per-project) - this works by > > > having > > > >>> self-hosted runners assigned per project (we needed infra JIRA > ticket > > > and > > > >>> generation of a bunch of tokens for our runners and our own AWS > > account > > > >>> with auto-scaling). > > > >>> > > > >>> It would be indeed great if it could be available from GitHub, but > so > > > >>> far we do not have any of those. > > > >>> > > > >>> J. > > > >>> > > > >>> > > > >>> > > > >>>> On Wed, Apr 7, 2021 at 9:31 PM Hyukjin Kwon <gurwls...@gmail.com> > > > >>>> wrote: > > > >>>> > > > >>>> > Thanks Martin for your feedback. > > > >>>> > > > > >>>> > > What was your reason to migrate from Apache Jenkins to Github > > > >>>> Actions ? > > > >>>> > > > > >>>> > I am sure there were more reasons for migrating from Amplap > > Jenkins > > > >>>> > <https://amplab.cs.berkeley.edu/jenkins/> to GitHub Actions but > > as > > > >>>> far as > > > >>>> > I can remember: > > > >>>> > - To reduce the maintenance cost of machines > > > >>>> > - The Jenkins machines became unstable and slow causing CI jobs > to > > > >>>> fail or > > > >>>> > be very flaky. > > > >>>> > - Difficulty to manage the installed libraries. > > > >>>> > - Intermittent unknown issues in the machines > > > >>>> > > > > >>>> > Yes, one option might be to consider other options to migrate > > again. > > > >>>> > However, other projects will very likely suffer the > > > >>>> > same problem. In addition, the migration in a large project is > not > > > an > > > >>>> > easy work to do > > > >>>> > > > > >>>> > I would like to know the feasibility of having more resources in > > > >>>> GitHub > > > >>>> > Actions, or, for example, having sub-groups where > > > >>>> > each group shares the resources - currently one GitHub > > organisation > > > >>>> shares > > > >>>> > all resources across the projects. > > > >>>> > > > > >>>> > > > > >>>> > 2021년 4월 7일 (수) 오후 10:04, Martin Grigorov <mgrigo...@apache.org > > >님이 > > > >>>> 작성: > > > >>>> > > > > >>>> >> > > > >>>> >> > > > >>>> >> On Wed, Apr 7, 2021 at 3:41 PM Hyukjin Kwon < > gurwls...@gmail.com > > > > > > >>>> wrote: > > > >>>> >> > > > >>>> >>> Hi Greg, > > > >>>> >>> > > > >>>> >>> I raised this thread to figure out a way that we can work > > together > > > >>>> to > > > >>>> >>> resolve this issue, gather feedback, and to understand how > other > > > >>>> projects > > > >>>> >>> work around. > > > >>>> >>> Several projects I observed, as far as I can tell, have made > > > enough > > > >>>> >>> efforts > > > >>>> >>> to save the resources in GitHub Actions but still suffer from > > the > > > >>>> lack of > > > >>>> >>> resources. > > > >>>> >>> > > > >>>> >> > > > >>>> >> And it will get even worse because: > > > >>>> >> 1) more and more Apache projects migrate from TravisCI to > Github > > > >>>> Actions > > > >>>> >> (GA) > > > >>>> >> 2) new projects join ASF and many of them already use GA > > > >>>> >> > > > >>>> >> > > > >>>> >> What was your reason to migrate from Apache Jenkins to Github > > > >>>> Actions ? > > > >>>> >> If you want dedicated resources then you will need to manage > the > > CI > > > >>>> >> yourself. > > > >>>> >> You could use Apache Jenkins/Buildbot with dedicated agents for > > > your > > > >>>> >> project. > > > >>>> >> Or you could set up your own CI infrastructure with Jenkins, > > > DroneIO, > > > >>>> >> ConcourceCI, ... > > > >>>> >> > > > >>>> >> Yet another option is to move to CircleCI or Cirrus. They are > > > >>>> similar to > > > >>>> >> TravisCI / GA and less crowded (for now). > > > >>>> >> > > > >>>> >> Martin > > > >>>> >> > > > >>>> >> I appreciate the resources provided to us but that does not > > resolve > > > >>>> the > > > >>>> >>> issue of the development being slowed down. > > > >>>> >>> > > > >>>> >>> > > > >>>> >>> 2021년 4월 7일 (수) 오후 5:52, Greg Stein <gst...@gmail.com>님이 작성: > > > >>>> >>> > > > >>>> >>> > On Wed, Apr 7, 2021 at 12:25 AM Hyukjin Kwon < > > > gurwls...@gmail.com > > > >>>> > > > > >>>> >>> wrote: > > > >>>> >>> > > > > >>>> >>> >> Hi all, > > > >>>> >>> >> > > > >>>> >>> >> I am an Apache Spark PMC, > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > You are a member of the Apache Spark PMC. You are *not* a > PMC. > > > >>>> Please > > > >>>> >>> stop > > > >>>> >>> > with that terminology. The Foundation has about 200 PMCs, > and > > > you > > > >>>> are a > > > >>>> >>> > member of one of them. You are NOT a "PMC" .. you're a > > person. A > > > >>>> PMC > > > >>>> >>> is a > > > >>>> >>> > construct of the Foundation. > > > >>>> >>> > > > > >>>> >>> > >... > > > >>>> >>> > > > > >>>> >>> >> I am aware of the limited GitHub Actions resources that are > > > >>>> shared > > > >>>> >>> >> across all projects in ASF, > > > >>>> >>> >> and many projects suffer from it. This issue significantly > > > slows > > > >>>> down > > > >>>> >>> the > > > >>>> >>> >> development cycle of > > > >>>> >>> >> other projects, at least Apache Spark. > > > >>>> >>> >> > > > >>>> >>> > > > > >>>> >>> > And the Foundation gets those build minutes for GitHub > Actions > > > >>>> >>> provided to > > > >>>> >>> > us from GitHub and Microsoft, and we are thankful that they > > > >>>> provide > > > >>>> >>> them to > > > >>>> >>> > the Foundation. Maybe it isn't all the build minutes that > > every > > > >>>> group > > > >>>> >>> > wants, but that is what we have. So it is incumbent upon all > > of > > > >>>> us to > > > >>>> >>> > figure out how to build more, with fewer minutes. > > > >>>> >>> > > > > >>>> >>> > Say "thank you" to GitHub, please. > > > >>>> >>> > > > > >>>> >>> > Regards, > > > >>>> >>> > -g > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > >>>> >> > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> +48 660 796 129 > > > >>> > > > >> > > > > > >
