Just dropping a comment. Apache Spark solved it by splitting the job. As of the number of parallel jobs, Apache Spark made, in PR builder, a custom logic to link the GitHub workflow run in forked repositories - so we reuse the GitHub resources in PR authors forked repository instead of the one allocated to ASF itself.
On Fri, Apr 14, 2023 at 8:00 AM sebb <seb...@gmail.com> wrote: > On Thu, 13 Apr 2023 at 20:58, Martin Grigorov <mgrigo...@apache.org> > wrote: > > > > Hi, > > > > On Thu, Apr 13, 2023 at 7:17 PM Sai Boorlagadda < > sai_boorlaga...@apache.org> > > wrote: > > > > > Hey All! I am part of Apache Geode project and we have been migrating > our > > > pipelines to Github actions and hit a roadblock that the max. job > execution > > > time on non-self-hosted GitHub workers is set a hard limit > > > < > > > > https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration > > > > > > > of > > > 6 hours and one of our job > > > <https://github.com/apache/geode/actions/runs/4639012912> is taking > more > > > than 6 hours. Are there any pointers on how someone solved this? or > does > > > > Github provides any increases for Apache Foundation projects? > > > > > > > The only way to "increase the resources" is to use a self-hosted runner. > > But instead of looking how to use more of the free pool you should try to > > optimize your build to need less! > > These free resources are shared with all other Apache projects, so when > > your project uses more another project will have to wait. > > > > You can start by using parallel build - > > > https://github.com/apache/geode/blob/102e24691eacd2d1d6652a070f14af9f5b42dc0d/.github/workflows/gradle.yml#L254 > > Also tune the maxWorkers - > > > https://github.com/apache/geode/blob/102e24691eacd2d1d6652a070f14af9f5b42dc0d/.github/workflows/gradle.yml#L256 > . > > The Linux VMs have 2 vCPUs. You can try with the macos-latest VM,it has 3 > > vCPUs. > > Another option is to split this job into few smaller ones. Each job has > its > > own 6 hours. > > Also maybe run some of the jobs manually, rather than on every commit. > At present there are two instances running at the same time from > subsequent commits. > At least one of these is a waste of resources. > > > Good luck! > > > > Martin >
