I realize our use case is a bit different than some of the big Java projects here, but I thought I'd give a few tips and tricks that have helped us along the way:
Allen Wittenauer <a...@effectivemachines.com.INVALID> said: > > On Jul 23, 2018, at 12:45 AM, Gavin McDonald > > <ga...@16degrees.com.au> wrote: > > > > Is there any reason at all to keep the 'workspace' dirs of builds > > on the > > jenkins slaves ? > > - Some jobs download and build external dependencies, using the > workspace directories as a cache and to avoid sending more work to > INFRA. Removing the cache may greatly increase build time, network > bandwidth, and potentially increase INFRA’s workload. This is why we switched to Docker for ASF Jenkins CI. By pre-building our Docker container images for CI, we take control over the build environment in a very proactive way, reducing Infra's investment to just keeping the build nodes up, running, and with sufficient disk space. The build environments supported on the build hosts in Jenkins seem very skewed towards Java dependencies (which is fine), and our build chain is entirely orthogonal (specific JavaScript libraries, Erlang versions, etc.) It also lets us do things like pre-clone the source tree into the image, so that only deltas need to be checked out on each new build that runs. The hiccup then becomes pulling that image down onto each build node at the start of each build step. We forcibly launch a `docker pull` command in our Jenkinsfile, since otherwise an update to our build image may get missed. If the `docker pull` fails, the build fails, but then again it would also fail in the no-Docker case if dependencies could not be downloaded and run locally. It also means that, once a build is done, there is no mess on the Jenkins build node to clean up - just a regular `docker rm` or `docker rmi` is sufficient to restore disk space. Infra is already running these aggressively, since if a build hangs due to an unresponsive docker daemon or network failure, our post-run script to clean up after ourselves may never run. > - Many jobs don’t put everything into the saved artifacts due to > size constraints. Removing the workspace will almost certainly > guarantee that artifact usage goes way up as the need to grab (or > cache) bits from the workspace will be impossible with an overly > aggressive workspace deletion policy. We don't put everything into saved artefacts either, but we have built a simple Apache CouchDB-based database to which we upload any artefacts we want to save for development purposes only. (Right now, we use this mechanism to save build logs and other files for failed runs only.) We run this service for ourselves on couchdb-vm2.a.o. We'd be happy to explain how you can use something similar in your own builds, if it would help reduce the burden on the main Jenkins infra. > Maven, ant, etc don’t perform directory locks on local repositories. > Separate storage areas for jars are key so that multiple executors > don’t step all over each other. This was a HUGE problem for a lot > of jobs when multiple executors were introduced a few years ago. We had this issue too - which is why we build under a `/tmp` directory inside the Docker container to avoid one build trashing another build's workspace directory via the multi-node sync mechanism. -Joan
