Thanks for pointing this out Giacomo :). I dived into the VM and found a few zombie gem5 build processes kicking around (all from Jan 15th), which i'm 85% sure were the reason everything had gotten so sluggish. I've killed them and things _appear_ to be running a bit more speedy. Let's see.
-- Dr. Bobby R. Bruce Room 2235, Kemper Hall, UC Davis Davis, CA, 95616 web: https://www.bobbybruce.net On Wed, Jan 27, 2021 at 2:39 PM Giacomo Travaglini < giacomo.travagl...@arm.com> wrote: > P.S. A lot of other builds would have failed using today's timeout. > Check for example build #176 and #177 of the Nightly (end of December) > > This makes a lot of sense as these days overlap with the Compiler-Checks > run. > In fact, almost every time we run Compiler-Checks the Nightly takes more > than a day to complete > > Kind Regards > > Giacomo > > > > -----Original Message----- > > From: Giacomo Travaglini via gem5-dev <gem5-dev@gem5.org> > > Sent: 27 January 2021 22:23 > > To: gem5 Developer List <gem5-dev@gem5.org> > > Cc: Giacomo Travaglini <giacomo.travagl...@arm.com> > > Subject: [gem5-dev] Re: Some evidence build times have substantially > > increased on Jenkins > > > > Hi Bobby, > > > > While I agree there might be something wrong with it, It can be partly > > explained with the sheer amount of patches which is being reviewed and > > merged in these days. This is affecting compilation time in the > following ways: > > > > a) Overloading Jenkins with the gem5-develop job -> this is not valid > anymore > > as it has been removed > > b) By changing big portions of code, increasing recompilation time > (there are > > more translation units to be recompiled) > > > > If I have a look at the Jenkins history for example, most of the > Nightlies at the > > beginning of the month had almost no patches > > > > Nighly #180 for example took 13 hours but as you can see had no patches > > > > https://jenkins.gem5.org/job/Nightly/180/ > > > > The first run that was testing more than two patches is #183 > > > > https://jenkins.gem5.org/job/Nightly/184/ > > > > And in fact it took 1 day and two hours to complete. This would have > failed > > with the current setup of timing out in 23 hours. > > If you compare #patches with execution time you see a strong correlation > > between these two > > > > Same applies with the compiler check. Now situation is to get a bit > better, but > > IMHO in the last few days the combination of a) + b) has been lethal. > > > > There might be something else though > > > > Kind Regards > > > > Giacomo > > > > > -----Original Message----- > > > From: Bobby Bruce via gem5-dev <gem5-dev@gem5.org> > > > Sent: 27 January 2021 21:00 > > > To: gem5 Developer List <gem5-dev@gem5.org> > > > Cc: Bobby Bruce <bbr...@ucdavis.edu> > > > Subject: [gem5-dev] Some evidence build times have substantially > > > increased on Jenkins > > > > > > Dear all, > > > > > > As you may have noticed from the mailing list, our nightly builds are > > > failing due to timeouts, as are the compiler checkers. Though this is > > > just a preliminary look at the problem, I've noticed that the gem5 > > > build times on our Jenkins have increased substantially. Looking at > > > our last passing compiler checks, gem5 compiled in roughly 1.5 hours: > > > http://jenkins.gem5.org/job/Compiler- > > > Checks/40/. Looking at our first failing compiler checks, this has > > > jumped to roughly 3.5 hours: http://jenkins.gem5.org/job/Compiler- > > Checks/41/. > > > > > > This is just an FYI and a reachout to see if anyone has any ideas why > > > this might be. I'm currently looking to see if I can replicate these > > > poor compile times outside of Jenkins. I have noticed that our Gerrit > > > Kokoro appears to have continued running just fine (no timeouts), so > > > this may be some issue specific to our Jenkins setup. > > > > > > > > > Kind regards, > > > Bobby > > > > > > -- > > > > > > Dr. Bobby R. Bruce > > > Room 2235, > > > Kemper Hall, UC Davis > > > Davis, > > > CA, 95616 > > > > > > > > > web: https://www.bobbybruce.net > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > > confidential and may also be privileged. If you are not the intended > recipient, > > please notify the sender immediately and do not disclose the contents to > any > > other person, use it for any purpose, or store or copy the information > in any > > medium. Thank you. > > _______________________________________________ > > gem5-dev mailing list -- gem5-dev@gem5.org To unsubscribe send an email > to > > gem5-dev- > > le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. >
_______________________________________________ gem5-dev mailing list -- gem5-dev@gem5.org To unsubscribe send an email to gem5-dev-le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
