On Thu, Apr 26, 2018, 16:42 Greg Stein <gst...@gmail.com> wrote: > On Thu, Apr 26, 2018, 15:06 Alex Harui <aha...@adobe.com.invalid> wrote: > >> Perhaps I wasn't clear. I am not concerned about Jenkins projects being >> removed, just the artifacts of those projects. I am trying to make two >> points: >> >> 1) Old artifacts might have value as a known good build for a job that >> may not get run for years. So please don't run a clean up that deletes all >> artifacts older than N days. >> > > Nope. Download those artifacts if you want to keep them. The Jenkins > Master is not an archival system. It exists for our projects to do builds. > We have identified the overabundance of old artifacts as detrimental to the > service, which affects everyone. >
To be clearer here, Chris earlier said/asked: we’d ask that this be exported and managed locally by the project rather than remaining “live” on the master. Is there a reason other than convenience for it to remain live? Cheers, -g > 2) Somebody else pointed this out as well, but there appear to be folders >> listed for which there is no current Jenkins job. >> > > Knew about that. I simply read your note as "please keep these jobs". Some > disabled, some not working, etc. It seemed you were talking about jobs > rather than artifacts within them. > > Cheers, > -g > > >> Thanks, >> -Alex >> >> On 4/26/18, 11:07 AM, "Greg Stein" <gst...@gmail.com> wrote: >> >> Note that jobs will remain. >> >> This is only about deleting build artifacts. >> >> On Thu, Apr 26, 2018 at 12:40 PM, Alex Harui <aha...@adobe.com.invalid >> > >> wrote: >> >> > HI Chris, >> > >> > Thanks for the list. >> > >> > I’m going through the Flex-related jobs and have some feedback: >> > >> > flex-blazeds (maven) We’ve kept this build around even though it >> hasn’t >> > run in a while in case we need to do another release of blazeds. I >> would >> > like to keep at least one known good build in case we have trouble >> > resurrecting it later if we need it, even though it may sit idle >> for years. >> > >> > flex-flexunit_1 >> > flex-sdk_1 >> > flex-sdk_pixelbender_1 >> > flex-sdk_release_1 >> > flex-tlf_1 >> > flex_sdk_version I cannot find a project with these names in >> Jenkins. So >> > feel free to toss it. >> > >> > >> > flex-flexunit (maven) This project was never completed to build >> > successfully, but it would be nice to keep it around in case we >> need it. >> > >> > FlexJS Compiler (maven) >> > FlexJS Framework (maven) >> > FlexJS Pipeline >> > FlexJS Typedefs (maven) Looks like we never set the build limit, >> so I >> > just did that. The project is disabled, we are keeping it around as >> > archival for Royale, so not sure it will clean itself up. >> > >> > flex-productdashboard I deleted this project. >> > >> > flex-tool-api (maven) >> > flex-sdk-converter (maven) I’m not seeing old artifacts in these >> > projects, but they may also sit idle for years until some bug needs >> fixing. >> > >> > Flex-Site (Maven) This project never took off, but again it would >> be nice >> > to keep it around in case it gets revived >> > >> > In sum, a project like Flex may have several kinds of “products” >> with >> > varying activity levels and thus may have jobs that are idle for >> years and >> > it can be helpful to keep at least the last build around as a >> reference in >> > case the next time we run the build there is a failure. Please >> notify us >> > if we miss limiting the number of old builds. I think I fixed the >> ones >> > that didn’t have limits. But there does seem to be folders left >> around for >> > builds I think we deleted. >> > >> > Thanks, >> > -Alex >> > >> > >> > From: Chris Lambertus <c...@apache.org> >> > Reply-To: <builds@apache.org> >> > Date: Wednesday, April 25, 2018 at 12:04 AM >> > To: <builds@apache.org> >> > Subject: Re: purging of old job artifacts >> > >> > >> > >> > >> > On Apr 24, 2018, at 8:04 PM, Allen Wittenauer < >> a...@effectivemachines.com< >> > mailto:a...@effectivemachines.com>> wrote: >> > >> > >> > >> > On Apr 24, 2018, at 5:01 PM, Greg Stein <gst...@gmail.com<mailto: >> gstei >> > n...@gmail.com>> wrote: >> > >> > Let's go back to the start: stuff older than six months will be >> deleted. >> > What could possibly need to be retained? >> > >> > - Not every job runs every day. Some are extremely >> > situational. >> > >> > The artifacts do not need to be kept in perpetuity. When every >> project >> > does this, there are significant costs in both disk space and >> performance. >> > Our policy has been 30 days or 10 jobs retention. >> > >> > >> > >> > >> > - Some users might have specifically marked certain >> data >> > to be retained for very specific reasons. >> > >> > I know in my case I marked some logs to not be >> deleted >> > because I was using them to debug the systemic Jenkins build node >> crashes. >> > I want to keep the data to see if the usage numbers, etc, go down >> over time. >> > >> > >> > Part of the systemic problems are due to copious amounts of >> historical >> > data which are loaded into jenkins on startup, inflating the memory >> usage >> > and startup times. Again, when every job does this, it adds up, and >> many of >> > the problems we’re facing appear to be rooted in the very large >> number of >> > artifacts we have. >> > >> > >> > >> > So yes, there may be some value to some of that >> data that >> > will not be obvious to an outside observer. >> > >> > >> > Assume all jobs will be touched. >> > >> > … which is why giving a directory listing of just >> the base >> > directory would be useful to see who needs to look. If INFRA is >> unwilling >> > to provide that data, then keep any directories that reference: >> > >> > >> > Please dispense with the passive aggressive “unwilling to provide” >> > nonsense. This is inflammatory and anti-Infra for no valid reason. >> This >> > process is meant to be a pragmatic approach to cleaning up and >> improving a >> > service used by a large number of projects. The fact that I didn’t >> have >> > time to post the job list in the 4 hours since my last reply does >> not need >> > to be construed as reticence on Infra’s part to provide it. >> > >> > The top-level list of jobs is available here: >> > >> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.apache.org%2Fr37e&data=02%7C01%7Caharui%40adobe.com%7C1099543a29ab494553cc08d5aba08f41%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636603628504178337&sdata=dZOVSreSfQIqbL%2FScGSEW5m93SmDcKeb2%2FZMbPaIUzA%3D&reserved=0 >> > >> > I am happy to provide further information, however, due to the disk >> IO >> > issues on jenkins-master and the size of the jobs/ dir, multiple >> scans and >> > data analytics are difficult to provide due to the timescale. >> > >> > >> > As I previously mentioned, the list of actual artifacts currently >> slated >> > for deletion is 590MB and took several hours to generate. I also >> misspoke >> > earlier, that list is for artifacts over one year old. The space >> which >> > would be freed up is over 480GB. The list of artifacts over 180 >> days old is >> > going to be much longer, but I can look into making it available >> somewhere. >> > I question the utility though, as the 1 year data is over 3 million >> lines. >> > >> > >> > >> > >> > - precommit >> > - hadoop >> > - yarn >> > - hdfs >> > - mapreduce >> > - hbase >> > - yetus >> > >> > >> > We will not be cherry-picking jobs to exclude from the purge unless >> there >> > is a compelling operational reason to do so. Jenkins is a shared >> resource, >> > and all projects are affected equally. >> > >> > >> > Let me do some further research and compare the size and file >> counts for >> > artifacts vs. build metadata (logs, etc.) >> > >> > The main things we want to purge are: >> > >> > - all artifacts and metadata where the job/project longer exists >> > - binary artifacts with no value older than 180 days >> > >> > and, to a lesser extent, jobs which fall outside our general 30 >> day/10 >> > jobs retention policy. >> > >> > >> > As an example of ancient binary artifacts, there are 22MB of >> javadocs from >> > 2013 in /x1/jenkins/jenkins-home/jobs/ManifoldCF-mvn >> > >> > Using the yetus jobs as a reference, yetus-java builds 480 and 481 >> are >> > nearly a year old, but only contain a few kilobytes of data. While >> removing >> > them saves no space, they also provide no value, but are still >> > loaded/parsed by jenkins. Since they don’t contain valid jenkins >> objects, >> > they don’t even show up in the build history, but are still part of >> the >> > constant scanning of the jobs/ directory that jenkins does, and >> contribute >> > to high load and disk IO. Those two are the only +180 day artifacts >> for >> > yetus with the exception of a zero-byte legacyIds file for -qbt. >> > >> > root@jenkins-master:/x1/jenkins/jenkins-home/jobs# find yetus-* >> -mtime >> > +180 -ls >> > 69210803 4 drwxr-xr-x 2 jenkins jenkins 4096 Jul 12 >> 2017 >> > yetus-java/builds/481 >> > 69210815 4 -rw-r--r-- 1 jenkins jenkins 457 Jul 8 >> 2017 >> > yetus-java/builds/481/polling.log >> > 65813999 0 lrwxrwxrwx 1 jenkins jenkins 2 May 23 >> 2016 >> > yetus-java/builds/lastUnstableBuild -> -1 >> > 65814012 0 -rw-r--r-- 1 jenkins jenkins 0 May 23 >> 2016 >> > yetus-java/builds/legacyIds >> > 69210796 4 drwxr-xr-x 2 jenkins jenkins 4096 Jul 12 >> 2017 >> > yetus-java/builds/480 >> > 69210810 4 -rw-r--r-- 1 jenkins jenkins 456 Jul 7 >> 2017 >> > yetus-java/builds/480/polling.log >> > 23725477 0 lrwxrwxrwx 1 jenkins jenkins 2 Jun 15 >> 2017 >> > yetus-qbt/builds/lastStableBuild -> -1 >> > 23741645 0 lrwxrwxrwx 1 jenkins jenkins 2 Apr 14 >> 2016 >> > yetus-qbt/builds/lastUnstableBuild -> -1 >> > 23725478 0 lrwxrwxrwx 1 jenkins jenkins 2 Jun 15 >> 2017 >> > yetus-qbt/builds/lastSuccessfulBuild -> -1 >> > 23741647 0 -rw-r--r-- 1 jenkins jenkins 0 Apr 14 >> 2016 >> > yetus-qbt/builds/legacyIds >> > >> > For mapreduce, there is an empty Mapreduce-Patch-vesta.apache.org< >> > >> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2FMapreduce-Patch-vesta.apache.org&data=02%7C01%7Caharui%40adobe.com%7C1099543a29ab494553cc08d5aba08f41%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636603628504178337&sdata=KxgEx5GbH1omcfeO7sw0to9qBk7eJ%2BY3zhuM4k%2FlyUA%3D&reserved=0> >> from 2010, and a bunch of jobs >> > from June 2017 for PreCommit-MAPREDUCE-Build (6999-7006.) Again, >> while they >> > take up very little space, they are still loaded into jenkins and >> scanned >> > by the threads which watch the jobs/ dir for changes. Multiply this >> times >> > 2381 top level job configs, and you can see why we’re hoping this >> type of >> > purge will help improve jenkins performance and the frequent >> crashing. >> > >> > >> > Since we are looking to move to expensive NVMe disks (nearly 4TB >> worth) we >> > also need to perform due diligence to insure that we are not >> migrating and >> > maintaining ancient data. >> > >> > -Chris >> > >> > >> > >> > >> >> >>
