Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-20 Thread Micah Kornfield
OK, I'll try to do a little bit more investigation to see if I can get some basic integration setup (probably won't have bandwidth for at least two weeks). On Sun, Mar 15, 2020 at 3:41 AM Antoine Pitrou wrote: > > Le 15/03/2020 à 04:57, Wes McKinney a écrit : > > On Sat, Mar 14, 2020, 10:52 PM M

Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-15 Thread Antoine Pitrou
Le 15/03/2020 à 04:57, Wes McKinney a écrit : > On Sat, Mar 14, 2020, 10:52 PM Micah Kornfield > wrote: > >> Hi Antoine, >> Could you clarify what you mean by: >> >>> Given our current resource utilization on Github Actions, it seems that >>> even a non-auto-scaling setup could be useful. >> >>

Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-14 Thread Wes McKinney
On Sat, Mar 14, 2020, 10:52 PM Micah Kornfield wrote: > Hi Antoine, > Could you clarify what you mean by: > > > Given our current resource utilization on Github Actions, it seems that > > even a non-auto-scaling setup could be useful. > > > I could interpret it in a couple of ways ... > I think

Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-14 Thread Micah Kornfield
Hi Antoine, Could you clarify what you mean by: > Given our current resource utilization on Github Actions, it seems that > even a non-auto-scaling setup could be useful. I could interpret it in a couple of ways ... Thanks, Micah On Fri, Mar 13, 2020 at 7:36 AM Antoine Pitrou wrote: > > Le 1

Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-13 Thread Antoine Pitrou
Le 13/03/2020 à 01:45, Brian Hulette a écrit : > * What kind of devops tooling would be appropriate to provision and > manage the instances, scaling up and down based on need? > * What CI/CD platform would be appropriate to dispatch work to the > cloud nodes (taking into consideration the high co

Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-12 Thread Brian Hulette
* What kind of devops tooling would be appropriate to provision and manage the instances, scaling up and down based on need? * What CI/CD platform would be appropriate to dispatch work to the cloud nodes (taking into consideration the high costs of sysadmin, and seeking to minimize nodes sitting un

Re: [DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-11 Thread Micah Kornfield
> > * Who's going to pay for it? Perhaps Amazon, Google, or Microsoft can > donate cloud compute credits to the project Google has offered a donation of GCP credits based on some estimates I made last year when we were facing Travis CI issues. I'm happy to try to do some integration work to help m

[DISCUSS] Leveraging cloud computing resources for Arrow test workloads

2020-03-11 Thread Wes McKinney
hi folks, There has periodically been a discussion about employing dedicated compute resources to serve our testing needs beyond what can be accomplished in free / public CI services like GitHub Actions, Appveyor, etc. For example: * Workloads requiring a CUDA-capable GPU * Tests requiring a lot