> > * Who's going to pay for it? Perhaps Amazon, Google, or Microsoft can > donate cloud compute credits to the project
Google has offered a donation of GCP credits based on some estimates I made last year when we were facing Travis CI issues. I'm happy to try to do some integration work to help make this happen. For the other questions, I'm happy to do some research, but also happy if someone else would like to take up the work here. I think one blocker in the past has been restrictions from Apache Infra, is there any documentation on what is and is not supported on that front? Thanks, Micah On Wed, Mar 11, 2020 at 3:17 PM Wes McKinney <wesmck...@gmail.com> wrote: > hi folks, > > There has periodically been a discussion about employing dedicated > compute resources to serve our testing needs beyond what can be > accomplished in free / public CI services like GitHub Actions, > Appveyor, etc. For example: > > * Workloads requiring a CUDA-capable GPU > * Tests requiring a lot of memory > * ARM architecture > > While physical machines can be hooked up to some CI/CD services like > Github Actions and Buildkite, I believe we should not be 100% > dependent on the availability of such hardware (the recent tornado in > Nashville is a good example of what can go wrong). > > At some point it will make sense to be able to provision cloud hosts > (either temporary spot instances or persistent nodes) to meet these > needs. This brings up several questions: > > * Who's going to pay for it? Perhaps Amazon, Google, or Microsoft can > donate cloud compute credits to the project > * What kind of devops tooling would be appropriate to provision and > manage the instances, scaling up and down based on need? > * What CI/CD platform would be appropriate to dispatch work to the > cloud nodes (taking into consideration the high costs of sysadmin, and > seeking to minimize nodes sitting unused)? > > This will probably take time to work out and there is significant > engineering involved in achieving any solution, but it would be good > to have all the options on the table with a frank analysis of the > pros/cons and costs (both in money and volunteer time) involved. > > Thanks, > Wes >
