Hi,

The Heron project has a build that will last for about 2 hours and 40
minutes on average.  It is a single Jenkins job that spins up two
different docker containers consecutively.  We only run this job to
generate artifacts for a release.  You can see the job here:
https://builds.apache.org/job/apache-heron-github-build-release-packages/lastBuild/
-  Josh

On Wed, Jan 22, 2020 at 7:53 PM Chris Lambertus <c...@apache.org> wrote:

>
>
> > On Jan 22, 2020, at 4:55 PM, Chris Lambertus <c...@apache.org> wrote:
> >
> > Folks,
> >
> > Over the last week or so we have received many reports of broken builds
> due to nodes out of resources. As noted in INFRA-19751, builds appear to
> fail yet continue to run, using up all available resources on a build node.
> >
> > I will be implementing a system to kill jenkins processes based on
> duration of run. My initial feeling is to kill any single process which has
> been running for longer than one hour real-time.
> >
> > I will also be implementing a system to kill/purge all docker containers
> which have been running for over 6 hours.
>
> Additionally, orphaned docker jobs are causing major resource contention.
> I will be adding a weekly job to docker system prune —all && service docker
> restart.
>
> -Chris
>
>
>
> >
> >
> > I am seeking input on these time limits, especially from those with
> larger builds. Is there any reason a -single process- or a docker container
> should run for more than 1 or 6 hours respectively?
> >
> > Thanks,
> > Chris
> > ASF Infra
> >
>
>

Reply via email to