Note that jobs will remain. This is only about deleting build artifacts.
On Thu, Apr 26, 2018 at 12:40 PM, Alex Harui <aha...@adobe.com.invalid> wrote: > HI Chris, > > Thanks for the list. > > I’m going through the Flex-related jobs and have some feedback: > > flex-blazeds (maven) We’ve kept this build around even though it hasn’t > run in a while in case we need to do another release of blazeds. I would > like to keep at least one known good build in case we have trouble > resurrecting it later if we need it, even though it may sit idle for years. > > flex-flexunit_1 > flex-sdk_1 > flex-sdk_pixelbender_1 > flex-sdk_release_1 > flex-tlf_1 > flex_sdk_version I cannot find a project with these names in Jenkins. So > feel free to toss it. > > > flex-flexunit (maven) This project was never completed to build > successfully, but it would be nice to keep it around in case we need it. > > FlexJS Compiler (maven) > FlexJS Framework (maven) > FlexJS Pipeline > FlexJS Typedefs (maven) Looks like we never set the build limit, so I > just did that. The project is disabled, we are keeping it around as > archival for Royale, so not sure it will clean itself up. > > flex-productdashboard I deleted this project. > > flex-tool-api (maven) > flex-sdk-converter (maven) I’m not seeing old artifacts in these > projects, but they may also sit idle for years until some bug needs fixing. > > Flex-Site (Maven) This project never took off, but again it would be nice > to keep it around in case it gets revived > > In sum, a project like Flex may have several kinds of “products” with > varying activity levels and thus may have jobs that are idle for years and > it can be helpful to keep at least the last build around as a reference in > case the next time we run the build there is a failure. Please notify us > if we miss limiting the number of old builds. I think I fixed the ones > that didn’t have limits. But there does seem to be folders left around for > builds I think we deleted. > > Thanks, > -Alex > > > From: Chris Lambertus <c...@apache.org> > Reply-To: <builds@apache.org> > Date: Wednesday, April 25, 2018 at 12:04 AM > To: <builds@apache.org> > Subject: Re: purging of old job artifacts > > > > > On Apr 24, 2018, at 8:04 PM, Allen Wittenauer <a...@effectivemachines.com< > mailto:a...@effectivemachines.com>> wrote: > > > > On Apr 24, 2018, at 5:01 PM, Greg Stein <gst...@gmail.com<mailto:gstei > n...@gmail.com>> wrote: > > Let's go back to the start: stuff older than six months will be deleted. > What could possibly need to be retained? > > - Not every job runs every day. Some are extremely > situational. > > The artifacts do not need to be kept in perpetuity. When every project > does this, there are significant costs in both disk space and performance. > Our policy has been 30 days or 10 jobs retention. > > > > > - Some users might have specifically marked certain data > to be retained for very specific reasons. > > I know in my case I marked some logs to not be deleted > because I was using them to debug the systemic Jenkins build node crashes. > I want to keep the data to see if the usage numbers, etc, go down over time. > > > Part of the systemic problems are due to copious amounts of historical > data which are loaded into jenkins on startup, inflating the memory usage > and startup times. Again, when every job does this, it adds up, and many of > the problems we’re facing appear to be rooted in the very large number of > artifacts we have. > > > > So yes, there may be some value to some of that data that > will not be obvious to an outside observer. > > > Assume all jobs will be touched. > > … which is why giving a directory listing of just the base > directory would be useful to see who needs to look. If INFRA is unwilling > to provide that data, then keep any directories that reference: > > > Please dispense with the passive aggressive “unwilling to provide” > nonsense. This is inflammatory and anti-Infra for no valid reason. This > process is meant to be a pragmatic approach to cleaning up and improving a > service used by a large number of projects. The fact that I didn’t have > time to post the job list in the 4 hours since my last reply does not need > to be construed as reticence on Infra’s part to provide it. > > The top-level list of jobs is available here: > https://paste.apache.org/r37e > > I am happy to provide further information, however, due to the disk IO > issues on jenkins-master and the size of the jobs/ dir, multiple scans and > data analytics are difficult to provide due to the timescale. > > > As I previously mentioned, the list of actual artifacts currently slated > for deletion is 590MB and took several hours to generate. I also misspoke > earlier, that list is for artifacts over one year old. The space which > would be freed up is over 480GB. The list of artifacts over 180 days old is > going to be much longer, but I can look into making it available somewhere. > I question the utility though, as the 1 year data is over 3 million lines. > > > > > - precommit > - hadoop > - yarn > - hdfs > - mapreduce > - hbase > - yetus > > > We will not be cherry-picking jobs to exclude from the purge unless there > is a compelling operational reason to do so. Jenkins is a shared resource, > and all projects are affected equally. > > > Let me do some further research and compare the size and file counts for > artifacts vs. build metadata (logs, etc.) > > The main things we want to purge are: > > - all artifacts and metadata where the job/project longer exists > - binary artifacts with no value older than 180 days > > and, to a lesser extent, jobs which fall outside our general 30 day/10 > jobs retention policy. > > > As an example of ancient binary artifacts, there are 22MB of javadocs from > 2013 in /x1/jenkins/jenkins-home/jobs/ManifoldCF-mvn > > Using the yetus jobs as a reference, yetus-java builds 480 and 481 are > nearly a year old, but only contain a few kilobytes of data. While removing > them saves no space, they also provide no value, but are still > loaded/parsed by jenkins. Since they don’t contain valid jenkins objects, > they don’t even show up in the build history, but are still part of the > constant scanning of the jobs/ directory that jenkins does, and contribute > to high load and disk IO. Those two are the only +180 day artifacts for > yetus with the exception of a zero-byte legacyIds file for -qbt. > > root@jenkins-master:/x1/jenkins/jenkins-home/jobs# find yetus-* -mtime > +180 -ls > 69210803 4 drwxr-xr-x 2 jenkins jenkins 4096 Jul 12 2017 > yetus-java/builds/481 > 69210815 4 -rw-r--r-- 1 jenkins jenkins 457 Jul 8 2017 > yetus-java/builds/481/polling.log > 65813999 0 lrwxrwxrwx 1 jenkins jenkins 2 May 23 2016 > yetus-java/builds/lastUnstableBuild -> -1 > 65814012 0 -rw-r--r-- 1 jenkins jenkins 0 May 23 2016 > yetus-java/builds/legacyIds > 69210796 4 drwxr-xr-x 2 jenkins jenkins 4096 Jul 12 2017 > yetus-java/builds/480 > 69210810 4 -rw-r--r-- 1 jenkins jenkins 456 Jul 7 2017 > yetus-java/builds/480/polling.log > 23725477 0 lrwxrwxrwx 1 jenkins jenkins 2 Jun 15 2017 > yetus-qbt/builds/lastStableBuild -> -1 > 23741645 0 lrwxrwxrwx 1 jenkins jenkins 2 Apr 14 2016 > yetus-qbt/builds/lastUnstableBuild -> -1 > 23725478 0 lrwxrwxrwx 1 jenkins jenkins 2 Jun 15 2017 > yetus-qbt/builds/lastSuccessfulBuild -> -1 > 23741647 0 -rw-r--r-- 1 jenkins jenkins 0 Apr 14 2016 > yetus-qbt/builds/legacyIds > > For mapreduce, there is an empty Mapreduce-Patch-vesta.apache.org< > http://Mapreduce-Patch-vesta.apache.org> from 2010, and a bunch of jobs > from June 2017 for PreCommit-MAPREDUCE-Build (6999-7006.) Again, while they > take up very little space, they are still loaded into jenkins and scanned > by the threads which watch the jobs/ dir for changes. Multiply this times > 2381 top level job configs, and you can see why we’re hoping this type of > purge will help improve jenkins performance and the frequent crashing. > > > Since we are looking to move to expensive NVMe disks (nearly 4TB worth) we > also need to perform due diligence to insure that we are not migrating and > maintaining ancient data. > > -Chris > > > >
