Hi, The Heron project has a build that will last for about 2 hours and 40 minutes on average. It is a single Jenkins job that spins up two different docker containers consecutively. We only run this job to generate artifacts for a release. You can see the job here: https://builds.apache.org/job/apache-heron-github-build-release-packages/lastBuild/
- Josh On Wed, Jan 22, 2020 at 7:53 PM Chris Lambertus <c...@apache.org> wrote: > > > > On Jan 22, 2020, at 4:55 PM, Chris Lambertus <c...@apache.org> wrote: > > > > Folks, > > > > Over the last week or so we have received many reports of broken builds > due to nodes out of resources. As noted in INFRA-19751, builds appear to > fail yet continue to run, using up all available resources on a build node. > > > > I will be implementing a system to kill jenkins processes based on > duration of run. My initial feeling is to kill any single process which has > been running for longer than one hour real-time. > > > > I will also be implementing a system to kill/purge all docker containers > which have been running for over 6 hours. > > Additionally, orphaned docker jobs are causing major resource contention. > I will be adding a weekly job to docker system prune —all && service docker > restart. > > -Chris > > > > > > > > > I am seeking input on these time limits, especially from those with > larger builds. Is there any reason a -single process- or a docker container > should run for more than 1 or 6 hours respectively? > > > > Thanks, > > Chris > > ASF Infra > > > >