Unfortunately, I am at Cape Canaveral, Kennedy Space Centre, and watching the Starlink Launch about that time (or at Atlantis Shuttle tour depending on the launch progress) so I will have to pass this time :(. But I'm looking forward to the result of the Github discussion afterwards.
I reviewed the Github Public roadmap ( https://github.com/orgs/github/projects/4247/views/1) and I have those questions for the GitHub team. Possibly we can send the questions to them in advance so that they can come better prepared. 1) The old one - metrics/tracking the jobs usage per project - Seems they are planning finally some "real" billing APIS eventually: https://github.com/github/roadmap/issues/246 (Planned for Q4 2022) but I think the scope of it described in the roadmap does not cover jobs usage in action "per project" - which is precisely what we need. So maybe we can ask them to include this information in the updated APIs? 2) Making self-hosted runners secure for public projects. This is what we currently do by semi-automatically patching https://github.com/actions/runner/pull/783 on top of the latest Runner. This is still not present in the roadmap from what I see, unfortunately (at least directly). The security of the public self-hosted runners increased somewhat by requiring approvals for new users, but it does not really solve all the security issues (actually - maybe proper implementation of 3) below - might provide more protection here (by proper isolation, one-time-use and possibly some total-resource-limiting in the auto-scaling option for self-hosted runners. Some more context on this one: This is what currently Github Actions Security advice on self-hosted runners is: https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security : > We recommend that you only use self-hosted runners with private repositories. This is because forks of your public repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow. > This is not an issue with GitHub-hosted runners because each GitHub-hosted runner is always a clean isolated virtual machine, and it is destroyed at the end of the job execution. Looking at the last statement above, it seems that proper isolation and resource monitoring provided by 3) below could make it "safe" to run self-hosted runners also for public repos. It would be great to listen what the Github Product team has to say about it. 3) Easy auto-scaling self-hosted setup for K8 with capability to scale to 0. Seems that as of 2 months it is actually planned in the Public Roadmap of Github https://github.com/github/roadmap/issues/555. The issue was added 2 months ago and it is planned for Q4 2022. Knowing that this is actually planned and will be released shortly, might actually make us put a break on what Apache Beam team implements on their side and our plans in Airflow to reuse this or some other 3rd-party K8S integration of self-hosted runners - as it seems what Github team plans is exactly what we need. Especially in case it will provide the isolation and security properties that could make it on-par with the Github Public runners (see question 2 above). Maybe then we could simply close the PR from Ash https://github.com/actions/runner/pull/783 and remove the need for auto-patching of the Runner for Airflow. 4) One of the last missing features for our workflows is the possibility of a job in workflow to wait for the result of another workflow (alternatively - capability to pause a workflow until it gets triggered from another workflow or manually). Currently it is impossible to have a workflow where a job waits for another job from another workflow. This could save us quite a bit of "active" waiting build time when we wait for images being built. I believe an equivalent feature has been planned some time ago in the roadmap, but I cannot find it any more. Would be great to see some statement from the GitHub product team about it. J. On Wed, Oct 19, 2022 at 3:03 PM Daan Hoogland <daan.hoogl...@gmail.com> wrote: > gavin et al, > Unfortunately I am going to be busy from 15:15 UTC till about 17:00 UTC. I > imagine the meeting will be over by the time I can join :( hope to be there > next time. > > On Mon, Oct 17, 2022 at 2:32 PM Gavin McDonald <gmcdon...@apache.org> > wrote: > > > Hi All, > > > > This Thursday 20th October at 1600 UTC. > > > > Add your name to the cwiki page if you intend to attend. > > > > > > > https://cwiki.apache.org/confluence/display/BUILDS/Builds+Agenda+--+2022-10-20 > > > > Note that I do have a Github meeting the following day, so I do not have > > answers for those seeking Metrics etc, but more questions to ask Github > > are welcome. > > > > Gav... > > > > -- > > > > *Gavin McDonald* > > Systems Administrator > > ASF Infrastructure Team > > > > > -- > Daan >
