I would like to propose the addition of a self-hosted runner system to the arrow repository to add speciality runners (arm64 and CUDA). This will allow us to compensate for the arm64 jobs that previously ran on Travis, which will be turned off EOY[1].
The migration to GitHub Issues will require a significant extension of our existing “comment bot”-workflows (e.g. assigning and labeling issues for non-committers, see [3]), with such a system we could add reserved runners that only pick up these “comment bot”-jobs to guarantee a smooth developer experience, regardless of the state of the ASF CI resources. As the allocation of GitHub-hosted runners for the Apache software foundation was recently increased, the queue times are currently low, but this will inevitably change and such a system would enable us to react quickly to such changes by adding new Windows and Linux nodes without any need for INFRA intervention. We at Voltron Data have been working on a Kubernetes based system to deploy auto-scaling ephemeral GitHub runners that can be seamlessly added to the arrow repository via a Github App. As the runners are ephemeral (each job is run in an isolated environment that is destroyed once the job is done) the usual security issues with self-hosted runners do not apply [2]. Voltron Data has open sourced the necessary Infrastructure as Code [4], this makes it possible for other interested parties to donate CI capacity to arrow or other ASF projects by cloning the IaC, setting up and maintaining their own Instance of the system. Voltron Data will set up and maintain one instance of the system. The dockerfiles for the runners will be added to the main arrow repo to facilitate easy changes and updates to the runner configuration for the community. Best, Jacob [1]: https://cwiki.apache.org/confluence/display/INFRA/Travis+Migrations [2]: https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security [3]: https://github.com/apache/arrow/actions/workflows/comment_bot.yml [4]: https://github.com/voltrondata-labs/gha-controller-infra
