Thanks Maureen. I had already checked this and our http://ci.xwiki.org/administrativeMonitor/OldData/manage is already empty.
Thanks -Vincent On 25 Aug 2014 at 14:24:13, Maureen Barger (mobar...@gmail.com(mailto:mobar...@gmail.com)) wrote: > FWIW we had that issue too and had to run manage old data > (administrativeMonitor/OldData/manage). > > > On Mon, Aug 25, 2014 at 8:11 AM, Vincent Massol wrote: > > > > > > > > On 25 Aug 2014 at 13:59:42, Baptiste Mathus > > (bmat...@batmat.net(mailto:bmat...@batmat.net)(mailto:bmat...@batmat.net)) > > wrote: > > > > > I know this sounds obvious, but did you do an upgrade recently? > > > > Yes but the problem is not new, we’ve been having it for a very long time, > > so it’s hard to know what the new version contributes to it. Actually the > > reason I upgraded was to try to fix the slowness issue ;) > > > > > If so, what was the versions of Jenkins/plugins before? Differences in > > > general? > > > > I don’t recall the previous version number. We’re now using 1.575 (I tried > > to upgrade to 1.576 but it broke too many things and I’m now waiting for > > 1.577 to try again). > > > > > If possible, you might then want to do a git bisect to try and identify > > > the offending commit(s). But sure, even more if 2 days are needed to > > > reproduce the issue, that's gonna take many days to tackle. > > > > That’s hard to do since our problem has been happening for a long time now. > > > > Some fresh news: > > > > * Today we’ve seen CPU usage at 800% for the jenkins master process. Note > > that we don’t do any build on the jenkins master, they’re all executed on > > slave agents. > > * Going to http://ci.xwiki.org/threadDump was showing a "Caused by: > > java.lang.OutOfMemoryError: GC overhead limit exceeded” error > > * We checked permgen and it was used at 99.72% > > * We were using -Xmx1024m -XX:MaxPermSize=384m on jenkins master and we’ve > > just increased the permgen to 512MB even though we don’t really understand > > why it would require so much permgen on jenkins master (for jenkins agents > > we’re using: -Xmx1536m -XX:MaxPermSize=512m -Xms512m and they seem ok). Ayn > > idea? > > * If we still have permgen issues with 512MB on master, we’ll take a heap > > dump and share it somewhere if we can’t find out the problem (likely since > > we don’t know the jenkins’s source code). > > > > Thanks > > -Vincent > > > > > Cheers > > > > > > > > > 2014-08-25 12:19 GMT+02:00 Vincent Massol : > > > > Thanks Baptiste. However I knew about the Disk usage plugin and we’ve > > > > removed it a long time ago :) > > > > > > > > FTR we’ve noticed that when we restart Jenkins its speed is ok for a 1 > > > > or 2 days and then it starts to slow down a lot till it becomes > > > > unusable. > > > > > > > > Thanks > > > > -Vincent > > > > > > > > > > > > On 25 Aug 2014 at 12:17:02, Baptiste Mathus > > > > (m...@batmat.net(mailto:m...@batmat.net)(mailto:m...@batmat.net)) wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Vincent, > > > > > Couldn't check your thread dumps just now since I'm currently > > > > > offline, but if you're using the disk usage plugin try removing it. > > > > > > > > > > > > > > > We removed it some weeks ago too after seeing perf issues and > > > > > analyzing out these dumps. And there's been quite a lot of people > > > > > here that seemed to get the same problem. > > > > > > > > > > > > > > > HTH > > > > > > > > > > > > > > > Cheers > > > > > > > > > > Le 23 août 2014 17:30, "Vincent Massol" a écrit : > > > > > > Hi guys, > > > > > > > > > > > > At XWiki (http://xwiki.org) we've been using Jenkins for a long > > > > > > time but we've had performance issues for a while. I've disabled > > > > > > the Dashboard plugin that we were using in order to track down the > > > > > > performance issues. But even after disabling we still have issues. > > > > > > > > > > > > It happens that regularly and often when we navigate to a job in > > > > > > the UI, our jenkins server takes over 10 minutes to answer. > > > > > > > > > > > > It happened to me just a few minutes ago at > > > > > > http://ci.xwiki.org/job/xwiki-platform-6.1.x/ and I was able to get > > > > > > 2 thread dumps, spaced by a few minutes: > > > > > > * Thread dump 1: > > > > > > https://gist.github.com/vmassol/361225e90113c49100b5 > > > > > > * Thread dump 2: > > > > > > https://gist.github.com/vmassol/e495d48b0dc0b14c52a0 > > > > > > > > > > > > Unfortunately I don't know Jenkin's code and it's hard for me to > > > > > > understand the problem. What I see is that a lot of threads seems > > > > > > to be doing hudson.remoting.FlightRecorderInputStream.read(). > > > > > > > > > > > > Any idea of what could be going on? > > > > > > > > > > > > Anything I could do to help debug the issue? > > > > > > > > > > > > Thanks a lot for any help > > > > > > -Vincent Massol > > > > > > XWiki Committer > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "Jenkins Users" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to > > > > > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)(mailto:jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)). > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > > > > You received this message because you are subscribed to a topic in > > > > > the Google Groups "Jenkins Users" group. > > > > > To unsubscribe from this topic, visit > > > > > https://groups.google.com/d/topic/jenkinsci-users/L6ZSESU5x6s/unsubscribe. > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)(mailto:jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)). > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "Jenkins Users" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to > > > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)(mailto:jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)). > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > -- > > > Baptiste MATHUS - http://batmat.net > > > Sauvez un arbre, > > > Mangez un castor ! -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "Jenkins Users" group. > > > To unsubscribe from this topic, visit > > > https://groups.google.com/d/topic/jenkinsci-users/L6ZSESU5x6s/unsubscribe. > > > To unsubscribe from this group and all its topics, send an email to > > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)(mailto:jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com)). > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Jenkins Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users%2bunsubscr...@googlegroups.com). > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/L6ZSESU5x6s/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users+unsubscr...@googlegroups.com). > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
