Re: broken builds taking up resources

2020-01-22 Thread Mick Semb Wever
The Cassandra dtest builds take ~12 hours. The unit tests over an hour We are looking into parallelising these, but work hasn't started on that yet. We recently parallelised a number of the unit test builds, and added pipeline builds, and subsequently builds have been crashing with full disks. Ye

Re: broken builds taking up resources

2020-01-22 Thread Martin Stockhammer
Hi, our average build time for the main archiva build job is about 1 hour on the apache build servers. We have a timeout of 2h configured in our pipeline. So, one hour is too short for us and we would appreciate, if you consider to increase your kill timeout to some higher value. Regards M

Re: broken builds taking up resources

2020-01-22 Thread Josh Fischer
Hi, The Heron project has a build that will last for about 2 hours and 40 minutes on average. It is a single Jenkins job that spins up two different docker containers consecutively. We only run this job to generate artifacts for a release. You can see the job here: https://builds.apache.org/job

Re: broken builds taking up resources

2020-01-22 Thread Chris Lambertus
> On Jan 22, 2020, at 4:55 PM, Chris Lambertus wrote: > > Folks, > > Over the last week or so we have received many reports of broken builds due > to nodes out of resources. As noted in INFRA-19751, builds appear to fail yet > continue to run, using up all available resources on a build nod

broken builds taking up resources

2020-01-22 Thread Chris Lambertus
Folks, Over the last week or so we have received many reports of broken builds due to nodes out of resources. As noted in INFRA-19751, builds appear to fail yet continue to run, using up all available resources on a build node. I will be implementing a system to kill jenkins processes based on

Re: H23/H26/H29/H41 job failures

2020-01-22 Thread Chesnay Schepler
Same issue occurred on H34: https://builds.apache.org/job/flink-snapshot-deployment-1.9/190/display/redirect On 21/01/2020 18:28, Chesnay Schepler wrote: For the past few days the Flink project has observed multiple out-of-memory errors on Jenkins:    H23: https://builds.apache.org/job/flink-