Hi All, Infra is working on self-hosted Github Runners provided by Infra to projects, hosted in Azure and hope to provide a few varieties of arch/cpu/mem.
Will keep this list updated as it progresses Gav... On Fri, Apr 14, 2023 at 9:23 AM Sai Boorlagadda <sai.boorlaga...@gmail.com> wrote: > thanks for all the feedback. At this time there are no sponsors for Geode > so cannot have self-hosted runners. I have already split the job running > tests into multiple jobs by gradle module, and even then one particular > gradle module takes more than 6 hours. Will try to parallelize and see if > that works. > > Sai > > On Thu, 13 Apr 2023 at 23:54, Jarek Potiuk <ja...@potiuk.com> wrote: > > > In many cases it can be done with choosing a bigger machine with more > CPUS > > and parallelising as others mentioned. This is cool if your tests are > pure > > unit tests and you can add just `--xdist` flag or similar (this is a > pytest > > extension to run your tests in parallel with as many CPUs as you can). > > However there are cases where the limitation is I/O or your tests simply > > cannot run in parallel because a lot of them rely on shared resources > (say > > database). But even then you can attempt to do something about it. > > > > In Airflow we solved those problems by custom-parallelising our jobs, > > choosing huge self-hosted runners and running everything in-memory. > > > > Even though our tests could not be parallelized "per tests" (mostly for > > historical reasons a lot of our tests are not pure unit tests and depend > on > > database), we split the tests into "test types" (8 of them but soon more) > > and run them in parallel - with as many parallel types running as we > have. > > Each test uses its own database instance - this is all orchestrated with > > docker-compose. > > In order to avoid inevitable I/O contention with this setup, this is all > > running on a huge tmpfs storage (50 GB or so) - including a docker > > instance that runs the databases that has tmpfs backing storage, so > those > > databases are backed by in-memory filesystem and thus are super-stable > and > > super-fast. Thanks to that, our thousands of tests can run really fast > > even if some of them are not pure unit tests. We run it all on a large > > self-hosted runner with 8 CPUS and 64 GB RAM and thanks to that our > > complete test suite runs in 15 minutes instead of 1.5 hour. > > > > Such setup achieves two optimisation goals: cheap and fast. Yes we need > > much more costly, bigger machines but we need them for a shorter time and > > we use them with 80%-90% utilisation which is pretty high for such cases > > (we keep optimising it regularly and I try to continue to push it closer > to > > 100% continuously). As the result - if your hosted runners in the cloud > are > > on-demand/ephemeral (usually 80%-90% cost reduction) and you have a fast > > setup, you can bring them up for 10 minutes and shutdown when finished, > > thus they cause a fraction of small machines that run all the time, > > especially if in the project you have times where no PRs are run. Also > > optimising speed of tests is even more important than optimising the cost > > of them, because getting feedback faster is good for your contributors - > > but with this setup we can eat cake and have it too - the cost is low and > > the tests are fast. > > > > J. > > > > > > > > On Fri, Apr 14, 2023 at 1:37 AM Hyukjin Kwon <gurwls...@gmail.com> > wrote: > > > > > Just dropping a comment. Apache Spark solved it by splitting the job. > > > > > > As of the number of parallel jobs, Apache Spark made, in PR builder, a > > > custom logic to link the GitHub workflow run in forked repositories - > so > > we > > > reuse the GitHub resources in PR authors forked repository instead of > the > > > one allocated to ASF itself. > > > > > > On Fri, Apr 14, 2023 at 8:00 AM sebb <seb...@gmail.com> wrote: > > > > > > > On Thu, 13 Apr 2023 at 20:58, Martin Grigorov <mgrigo...@apache.org> > > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > On Thu, Apr 13, 2023 at 7:17 PM Sai Boorlagadda < > > > > sai_boorlaga...@apache.org> > > > > > wrote: > > > > > > > > > > > Hey All! I am part of Apache Geode project and we have been > > migrating > > > > our > > > > > > pipelines to Github actions and hit a roadblock that the max. job > > > > execution > > > > > > time on non-self-hosted GitHub workers is set a hard limit > > > > > > < > > > > > > > > > > > > > > > > https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration > > > > > > > > > > > > > of > > > > > > 6 hours and one of our job > > > > > > <https://github.com/apache/geode/actions/runs/4639012912> is > > taking > > > > more > > > > > > than 6 hours. Are there any pointers on how someone solved this? > or > > > > does > > > > > > > > > > Github provides any increases for Apache Foundation projects? > > > > > > > > > > > > > > > > The only way to "increase the resources" is to use a self-hosted > > > runner. > > > > > But instead of looking how to use more of the free pool you should > > try > > > to > > > > > optimize your build to need less! > > > > > These free resources are shared with all other Apache projects, so > > when > > > > > your project uses more another project will have to wait. > > > > > > > > > > You can start by using parallel build - > > > > > > > > > > > > > > > https://github.com/apache/geode/blob/102e24691eacd2d1d6652a070f14af9f5b42dc0d/.github/workflows/gradle.yml#L254 > > > > > Also tune the maxWorkers - > > > > > > > > > > > > > > > https://github.com/apache/geode/blob/102e24691eacd2d1d6652a070f14af9f5b42dc0d/.github/workflows/gradle.yml#L256 > > > > . > > > > > The Linux VMs have 2 vCPUs. You can try with the macos-latest VM,it > > > has 3 > > > > > vCPUs. > > > > > Another option is to split this job into few smaller ones. Each job > > has > > > > its > > > > > own 6 hours. > > > > > > > > Also maybe run some of the jobs manually, rather than on every > commit. > > > > At present there are two instances running at the same time from > > > > subsequent commits. > > > > At least one of these is a waste of resources. > > > > > > > > > Good luck! > > > > > > > > > > Martin > > > > > > > > > > -- *Gavin McDonald* Systems Administrator ASF Infrastructure Team
