These are awesome! Has there been any luck in reaching out to IBM to see if they could donate one or more s390x VMs to use as runners for testing the s390x builds? That is probably my only concern with Travis going away at EOY, since we don't have a way currently to test those builds on GH Actions.
--Matt On Fri, Dec 16, 2022 at 8:46 AM Jacob Wujciak <ja...@voltrondata.com.invalid> wrote: > I would like to propose the addition of a self-hosted runner system to the > arrow repository to add speciality runners (arm64 and CUDA). This will > allow us to compensate for the arm64 jobs that previously ran on Travis, > which will be turned off EOY[1]. > > The migration to GitHub Issues will require a significant extension of our > existing “comment bot”-workflows (e.g. assigning and labeling issues for > non-committers, see [3]), with such a system we could add reserved runners > that only pick up these “comment bot”-jobs to guarantee a smooth developer > experience, regardless of the state of the ASF CI resources. > > As the allocation of GitHub-hosted runners for the Apache software > foundation was recently increased, the queue times are currently low, but > this will inevitably change and such a system would enable us to react > quickly to such changes by adding new Windows and Linux nodes without any > need for INFRA intervention. > > We at Voltron Data have been working on a Kubernetes based system to deploy > auto-scaling ephemeral GitHub runners that can be seamlessly added to the > arrow repository via a Github App. As the runners are ephemeral (each job > is run in an isolated environment that is destroyed once the job is done) > the usual security issues with self-hosted runners do not apply [2]. > > Voltron Data has open sourced the necessary Infrastructure as Code [4], > this makes it possible for other interested parties to donate CI capacity > to arrow or other ASF projects by cloning the IaC, setting up and > maintaining their own Instance of the system. Voltron Data will set up and > maintain one instance of the system. > > The dockerfiles for the runners will be added to the main arrow repo to > facilitate easy changes and updates to the runner configuration for the > community. > > Best, > Jacob > > [1]: https://cwiki.apache.org/confluence/display/INFRA/Travis+Migrations > > [2]: > > https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security > > [3]: https://github.com/apache/arrow/actions/workflows/comment_bot.yml > > [4]: https://github.com/voltrondata-labs/gha-controller-infra >
