On 25 Aug 2014 at 13:59:42, Baptiste Mathus
(bmat...@batmat.net(mailto:bmat...@batmat.net)) wrote:
> I know this sounds obvious, but did you do an upgrade recently?
Yes but the problem is not new, we’ve been having it for a very long time, so
it’s hard to know what the new version contributes to it. Actually the reason I
upgraded was to try to fix the slowness issue ;)
> If so, what was the versions of Jenkins/plugins before? Differences in
> general?
I don’t recall the previous version number. We’re now using 1.575 (I tried to
upgrade to 1.576 but it broke too many things and I’m now waiting for 1.577 to
try again).
> If possible, you might then want to do a git bisect to try and identify the
> offending commit(s). But sure, even more if 2 days are needed to reproduce
> the issue, that's gonna take many days to tackle.
That’s hard to do since our problem has been happening for a long time now.
Some fresh news:
* Today we’ve seen CPU usage at 800% for the jenkins master process. Note that
we don’t do any build on the jenkins master, they’re all executed on slave
agents.
* Going to http://ci.xwiki.org/threadDump was showing a "Caused by:
java.lang.OutOfMemoryError: GC overhead limit exceeded” error
* We checked permgen and it was used at 99.72%
* We were using -Xmx1024m -XX:MaxPermSize=384m on jenkins master and we’ve
just increased the permgen to 512MB even though we don’t really understand why
it would require so much permgen on jenkins master (for jenkins agents we’re
using: -Xmx1536m -XX:MaxPermSize=512m -Xms512m and they seem ok). Ayn idea?
* If we still have permgen issues with 512MB on master, we’ll take a heap dump
and share it somewhere if we can’t find out the problem (likely since we don’t
know the jenkins’s source code).
Thanks
-Vincent
> Cheers
>
>
> 2014-08-25 12:19 GMT+02:00 Vincent Massol :
> > Thanks Baptiste. However I knew about the Disk usage plugin and we’ve
> > removed it a long time ago :)
> >
> > FTR we’ve noticed that when we restart Jenkins its speed is ok for a 1 or 2
> > days and then it starts to slow down a lot till it becomes unusable.
> >
> > Thanks
> > -Vincent
> >
> >
> > On 25 Aug 2014 at 12:17:02, Baptiste Mathus
> > (m...@batmat.net(mailto:m...@batmat.net)) wrote:
> >
> >
> >
> > >
> > > Hi Vincent,
> > > Couldn't check your thread dumps just now since I'm currently offline,
> > > but if you're using the disk usage plugin try removing it.
> > >
> > >
> > > We removed it some weeks ago too after seeing perf issues and analyzing
> > > out these dumps. And there's been quite a lot of people here that seemed
> > > to get the same problem.
> > >
> > >
> > > HTH
> > >
> > >
> > > Cheers
> > >
> > > Le 23 août 2014 17:30, "Vincent Massol" a écrit :
> > > > Hi guys,
> > > >
> > > > At XWiki (http://xwiki.org) we've been using Jenkins for a long time
> > > > but we've had performance issues for a while. I've disabled the
> > > > Dashboard plugin that we were using in order to track down the
> > > > performance issues. But even after disabling we still have issues.
> > > >
> > > > It happens that regularly and often when we navigate to a job in the
> > > > UI, our jenkins server takes over 10 minutes to answer.
> > > >
> > > > It happened to me just a few minutes ago at
> > > > http://ci.xwiki.org/job/xwiki-platform-6.1.x/ and I was able to get 2
> > > > thread dumps, spaced by a few minutes:
> > > > * Thread dump 1: https://gist.github.com/vmassol/361225e90113c49100b5
> > > > * Thread dump 2: https://gist.github.com/vmassol/e495d48b0dc0b14c52a0
> > > >
> > > > Unfortunately I don't know Jenkin's code and it's hard for me to
> > > > understand the problem. What I see is that a lot of threads seems to be
> > > > doing hudson.remoting.FlightRecorderInputStream.read().
> > > >
> > > > Any idea of what could be going on?
> > > >
> > > > Anything I could do to help debug the issue?
> > > >
> > > > Thanks a lot for any help
> > > > -Vincent Massol
> > > > XWiki Committer
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "Jenkins Users" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to
> > > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users+unsubscr...@googlegroups.com).
> > > > For more options, visit https://groups.google.com/d/optout.
> > > --
> > > You received this message because you are subscribed to a topic in the
> > > Google Groups "Jenkins Users" group.
> > > To unsubscribe from this topic, visit
> > > https://groups.google.com/d/topic/jenkinsci-users/L6ZSESU5x6s/unsubscribe.
> > > To unsubscribe from this group and all its topics, send an email to
> > > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users+unsubscr...@googlegroups.com).
> > >
> > >
> > > For more options, visit https://groups.google.com/d/optout.
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Jenkins Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to
> > jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users+unsubscr...@googlegroups.com).
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Baptiste MATHUS - http://batmat.net
> Sauvez un arbre,
> Mangez un castor ! --
> You received this message because you are subscribed to a topic in the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/L6ZSESU5x6s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-users+unsubscr...@googlegroups.com(mailto:jenkinsci-users+unsubscr...@googlegroups.com).
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
