Would it make sense to periodically restart the VM to clear out stuff like this?
Gabe On Thu, Jan 28, 2021 at 12:31 PM Bobby Bruce via gem5-dev <gem5-dev@gem5.org> wrote: > Thanks for pointing this out Giacomo :). > > I dived into the VM and found a few zombie gem5 build processes kicking > around (all from Jan 15th), which i'm 85% sure were the reason everything > had gotten so sluggish. I've killed them and things _appear_ to be running > a bit more speedy. Let's see. > > -- > Dr. Bobby R. Bruce > Room 2235, > Kemper Hall, UC Davis > Davis, > CA, 95616 > > web: https://www.bobbybruce.net > > > On Wed, Jan 27, 2021 at 2:39 PM Giacomo Travaglini < > giacomo.travagl...@arm.com> wrote: > >> P.S. A lot of other builds would have failed using today's timeout. >> Check for example build #176 and #177 of the Nightly (end of December) >> >> This makes a lot of sense as these days overlap with the Compiler-Checks >> run. >> In fact, almost every time we run Compiler-Checks the Nightly takes more >> than a day to complete >> >> Kind Regards >> >> Giacomo >> >> >> > -----Original Message----- >> > From: Giacomo Travaglini via gem5-dev <gem5-dev@gem5.org> >> > Sent: 27 January 2021 22:23 >> > To: gem5 Developer List <gem5-dev@gem5.org> >> > Cc: Giacomo Travaglini <giacomo.travagl...@arm.com> >> > Subject: [gem5-dev] Re: Some evidence build times have substantially >> > increased on Jenkins >> > >> > Hi Bobby, >> > >> > While I agree there might be something wrong with it, It can be partly >> > explained with the sheer amount of patches which is being reviewed and >> > merged in these days. This is affecting compilation time in the >> following ways: >> > >> > a) Overloading Jenkins with the gem5-develop job -> this is not valid >> anymore >> > as it has been removed >> > b) By changing big portions of code, increasing recompilation time >> (there are >> > more translation units to be recompiled) >> > >> > If I have a look at the Jenkins history for example, most of the >> Nightlies at the >> > beginning of the month had almost no patches >> > >> > Nighly #180 for example took 13 hours but as you can see had no patches >> > >> > https://jenkins.gem5.org/job/Nightly/180/ >> > >> > The first run that was testing more than two patches is #183 >> > >> > https://jenkins.gem5.org/job/Nightly/184/ >> > >> > And in fact it took 1 day and two hours to complete. This would have >> failed >> > with the current setup of timing out in 23 hours. >> > If you compare #patches with execution time you see a strong correlation >> > between these two >> > >> > Same applies with the compiler check. Now situation is to get a bit >> better, but >> > IMHO in the last few days the combination of a) + b) has been lethal. >> > >> > There might be something else though >> > >> > Kind Regards >> > >> > Giacomo >> > >> > > -----Original Message----- >> > > From: Bobby Bruce via gem5-dev <gem5-dev@gem5.org> >> > > Sent: 27 January 2021 21:00 >> > > To: gem5 Developer List <gem5-dev@gem5.org> >> > > Cc: Bobby Bruce <bbr...@ucdavis.edu> >> > > Subject: [gem5-dev] Some evidence build times have substantially >> > > increased on Jenkins >> > > >> > > Dear all, >> > > >> > > As you may have noticed from the mailing list, our nightly builds are >> > > failing due to timeouts, as are the compiler checkers. Though this is >> > > just a preliminary look at the problem, I've noticed that the gem5 >> > > build times on our Jenkins have increased substantially. Looking at >> > > our last passing compiler checks, gem5 compiled in roughly 1.5 hours: >> > > http://jenkins.gem5.org/job/Compiler- >> > > Checks/40/. Looking at our first failing compiler checks, this has >> > > jumped to roughly 3.5 hours: http://jenkins.gem5.org/job/Compiler- >> > Checks/41/. >> > > >> > > This is just an FYI and a reachout to see if anyone has any ideas why >> > > this might be. I'm currently looking to see if I can replicate these >> > > poor compile times outside of Jenkins. I have noticed that our Gerrit >> > > Kokoro appears to have continued running just fine (no timeouts), so >> > > this may be some issue specific to our Jenkins setup. >> > > >> > > >> > > Kind regards, >> > > Bobby >> > > >> > > -- >> > > >> > > Dr. Bobby R. Bruce >> > > Room 2235, >> > > Kemper Hall, UC Davis >> > > Davis, >> > > CA, 95616 >> > > >> > > >> > > web: https://www.bobbybruce.net >> > >> > IMPORTANT NOTICE: The contents of this email and any attachments are >> > confidential and may also be privileged. If you are not the intended >> recipient, >> > please notify the sender immediately and do not disclose the contents >> to any >> > other person, use it for any purpose, or store or copy the information >> in any >> > medium. Thank you. >> > _______________________________________________ >> > gem5-dev mailing list -- gem5-dev@gem5.org To unsubscribe send an >> email to >> > gem5-dev- >> > le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> > _______________________________________________ > gem5-dev mailing list -- gem5-dev@gem5.org > To unsubscribe send an email to gem5-dev-le...@gem5.org > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
_______________________________________________ gem5-dev mailing list -- gem5-dev@gem5.org To unsubscribe send an email to gem5-dev-le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
