Just for a note, I already attempted to interact with GitHub team about resources problems a long ago with tagging infra team, and the infra team informed me that all ASF issues have to be discussed with them and addressed by them if I am remembering correctly.
It would be great to understand what has changed if I am supposed to independently discuss with the GitHub team to address this now. On Fri, 16 Apr 2021, 22:00 Martin Grigorov, <mgrigo...@apache.org> wrote: > > > On Fri, Apr 16, 2021 at 3:44 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: > >> I thought Jarek was pretty clear on that. I meant this: >> >> > So it all has to start with 'per-project' resource limitation and self- >> > budgeting. It would be GREAT if infra.could provide self-hosted GitHub >> > Runners SERVICE per project, where project could donate credits or money >> > for their own account, then the projects would have incentive to >> optimize >> > their own usage. I imagine this would be the best thing since the sliced >> > bread that INFRA could provide to all the projects. >> >> Maintaining and providing a self-hosted runners in GitHub Actions where >> the resources are managed in project level where each project can donate >> credits. >> >> In addition, Jarek mentioned that Airflow already has a working version - >> is it correct Jarek? >> >> If the infra team takes and improves it for other ASF projects, that >> would permanently resolve this issue. >> >> This suggestion looks reasonable and realistic to me. >> >> How do you think about this? >> > > I'll let Infra team respond for themselves but to me all these > improvements should be done by Github Actions team, not by each and every > project out there. > But if your project wants to use Apache Airflow's modifications then you > can do it - just follow what they did at > https://github.com/apache/airflow/blob/master/.github/workflows/ci.yml > > >> >> >> On Fri, 16 Apr 2021, 21:36 Martin Grigorov, <mgrigo...@apache.org> wrote: >> >>> Hi Hyukjin, >>> >>> On Fri, Apr 16, 2021 at 3:04 AM Hyukjin Kwon <gurwls...@gmail.com> >>> wrote: >>> >>> > Hi all, >>> > >>> > Is here the right place to expect feedback from the infra team or >>> related >>> > people? >>> > It would be great to hear what the infra team thinks about Jarek's >>> > suggestion. >>> > >>> >>> What suggestion exactly do you mean ? >>> I've just re-read Jarek's email and I see 3 tasks for Github Actions >>> team, >>> but nothing specific for Apache Infra team. >>> >>> >>> > >>> > >>> > 2021년 4월 13일 (화) 오전 11:15, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: >>> > >>> >> Hi all, >>> >> >>> >> Could we have any update and feedback from the INFRA team about >>> Jarek's >>> >> suggestion please? >>> >> >>> >> 2021년 4월 9일 (금) 오전 7:06, Jarek Potiuk <ja...@potiuk.com>님이 작성: >>> >> >>> >>> >>> >>>> That's a good idea. We do need to thank Github to give free >>> resources to >>> >>>> ASF projects, but it's better if we can make it a business: we allow >>> >>>> individual projects to sign deals with Github to get dedicated >>> >>>> resources. >>> >>>> It's a bit wasteful to ask every project to set up its own dev ops, >>> >>>> using Github Action is more convenient. Maybe we should raise it to >>> >>>> Github? >>> >>>> >>> >>> >>> >>> I do not think you can get per-project resources in GH - the most you >>> >>> can do are self-hosted runners for your project. >>> >>> >>> >>> (BTW I am not from the INFRA team - just a humble "CI person" of >>> Apache >>> >>> Airflow but very much vested into Github Actions) >>> >>> maybe the infra team can chime in here. We did raise it to GitHub, we >>> >>> even had meeting with them >>> >>> organized by Gavin and several topics were raised that could be >>> >>> eventually addressed by Github: >>> >>> >>> >>> - observability (they could not give us per-project usage dashboard >>> - we >>> >>> built our own imperfect (with API limitations) one by Tobiasz from >>> Airllow >>> >>> - security (limiting access to only project committers) - this we >>> >>> handled by the Ash's fork of Runner (but it's also imperfect - even >>> today I >>> >>> had to fix a problem where we had list of committers desynchronised >>> between >>> >>> our infra/CI.yml) >>> >>> - manageability (assigning resources per-project) - this works by >>> having >>> >>> self-hosted runners assigned per project (we needed infra JIRA >>> ticket and >>> >>> generation of a bunch of tokens for our runners and our own AWS >>> account >>> >>> with auto-scaling). >>> >>> >>> >>> It would be indeed great if it could be available from GitHub, but so >>> >>> far we do not have any of those. >>> >>> >>> >>> J. >>> >>> >>> >>> >>> >>> >>> >>>> On Wed, Apr 7, 2021 at 9:31 PM Hyukjin Kwon <gurwls...@gmail.com> >>> >>>> wrote: >>> >>>> >>> >>>> > Thanks Martin for your feedback. >>> >>>> > >>> >>>> > > What was your reason to migrate from Apache Jenkins to Github >>> >>>> Actions ? >>> >>>> > >>> >>>> > I am sure there were more reasons for migrating from Amplap >>> Jenkins >>> >>>> > <https://amplab.cs.berkeley.edu/jenkins/> to GitHub Actions but >>> as >>> >>>> far as >>> >>>> > I can remember: >>> >>>> > - To reduce the maintenance cost of machines >>> >>>> > - The Jenkins machines became unstable and slow causing CI jobs to >>> >>>> fail or >>> >>>> > be very flaky. >>> >>>> > - Difficulty to manage the installed libraries. >>> >>>> > - Intermittent unknown issues in the machines >>> >>>> > >>> >>>> > Yes, one option might be to consider other options to migrate >>> again. >>> >>>> > However, other projects will very likely suffer the >>> >>>> > same problem. In addition, the migration in a large project is >>> not an >>> >>>> > easy work to do >>> >>>> > >>> >>>> > I would like to know the feasibility of having more resources in >>> >>>> GitHub >>> >>>> > Actions, or, for example, having sub-groups where >>> >>>> > each group shares the resources - currently one GitHub >>> organisation >>> >>>> shares >>> >>>> > all resources across the projects. >>> >>>> > >>> >>>> > >>> >>>> > 2021년 4월 7일 (수) 오후 10:04, Martin Grigorov <mgrigo...@apache.org >>> >님이 >>> >>>> 작성: >>> >>>> > >>> >>>> >> >>> >>>> >> >>> >>>> >> On Wed, Apr 7, 2021 at 3:41 PM Hyukjin Kwon <gurwls...@gmail.com >>> > >>> >>>> wrote: >>> >>>> >> >>> >>>> >>> Hi Greg, >>> >>>> >>> >>> >>>> >>> I raised this thread to figure out a way that we can work >>> together >>> >>>> to >>> >>>> >>> resolve this issue, gather feedback, and to understand how other >>> >>>> projects >>> >>>> >>> work around. >>> >>>> >>> Several projects I observed, as far as I can tell, have made >>> enough >>> >>>> >>> efforts >>> >>>> >>> to save the resources in GitHub Actions but still suffer from >>> the >>> >>>> lack of >>> >>>> >>> resources. >>> >>>> >>> >>> >>>> >> >>> >>>> >> And it will get even worse because: >>> >>>> >> 1) more and more Apache projects migrate from TravisCI to Github >>> >>>> Actions >>> >>>> >> (GA) >>> >>>> >> 2) new projects join ASF and many of them already use GA >>> >>>> >> >>> >>>> >> >>> >>>> >> What was your reason to migrate from Apache Jenkins to Github >>> >>>> Actions ? >>> >>>> >> If you want dedicated resources then you will need to manage the >>> CI >>> >>>> >> yourself. >>> >>>> >> You could use Apache Jenkins/Buildbot with dedicated agents for >>> your >>> >>>> >> project. >>> >>>> >> Or you could set up your own CI infrastructure with Jenkins, >>> DroneIO, >>> >>>> >> ConcourceCI, ... >>> >>>> >> >>> >>>> >> Yet another option is to move to CircleCI or Cirrus. They are >>> >>>> similar to >>> >>>> >> TravisCI / GA and less crowded (for now). >>> >>>> >> >>> >>>> >> Martin >>> >>>> >> >>> >>>> >> I appreciate the resources provided to us but that does not >>> resolve >>> >>>> the >>> >>>> >>> issue of the development being slowed down. >>> >>>> >>> >>> >>>> >>> >>> >>>> >>> 2021년 4월 7일 (수) 오후 5:52, Greg Stein <gst...@gmail.com>님이 작성: >>> >>>> >>> >>> >>>> >>> > On Wed, Apr 7, 2021 at 12:25 AM Hyukjin Kwon < >>> gurwls...@gmail.com >>> >>>> > >>> >>>> >>> wrote: >>> >>>> >>> > >>> >>>> >>> >> Hi all, >>> >>>> >>> >> >>> >>>> >>> >> I am an Apache Spark PMC, >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > You are a member of the Apache Spark PMC. You are *not* a PMC. >>> >>>> Please >>> >>>> >>> stop >>> >>>> >>> > with that terminology. The Foundation has about 200 PMCs, and >>> you >>> >>>> are a >>> >>>> >>> > member of one of them. You are NOT a "PMC" .. you're a >>> person. A >>> >>>> PMC >>> >>>> >>> is a >>> >>>> >>> > construct of the Foundation. >>> >>>> >>> > >>> >>>> >>> > >... >>> >>>> >>> > >>> >>>> >>> >> I am aware of the limited GitHub Actions resources that are >>> >>>> shared >>> >>>> >>> >> across all projects in ASF, >>> >>>> >>> >> and many projects suffer from it. This issue significantly >>> slows >>> >>>> down >>> >>>> >>> the >>> >>>> >>> >> development cycle of >>> >>>> >>> >> other projects, at least Apache Spark. >>> >>>> >>> >> >>> >>>> >>> > >>> >>>> >>> > And the Foundation gets those build minutes for GitHub Actions >>> >>>> >>> provided to >>> >>>> >>> > us from GitHub and Microsoft, and we are thankful that they >>> >>>> provide >>> >>>> >>> them to >>> >>>> >>> > the Foundation. Maybe it isn't all the build minutes that >>> every >>> >>>> group >>> >>>> >>> > wants, but that is what we have. So it is incumbent upon all >>> of >>> >>>> us to >>> >>>> >>> > figure out how to build more, with fewer minutes. >>> >>>> >>> > >>> >>>> >>> > Say "thank you" to GitHub, please. >>> >>>> >>> > >>> >>>> >>> > Regards, >>> >>>> >>> > -g >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> >>> >>>> >> >>> >>>> >>> >>> >>> >>> >>> >>> -- >>> >>> +48 660 796 129 >>> >>> >>> >> >>> >>
