I just see these: Feb 6, 2013 9:50:10 PM hudson.plugins.im.IMConnectionProvider$ConnectorRunnable run INFO: Reconnect failed. Next connection attempt in 16 minutes Feb 6, 2013 9:51:40 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect dmwm-agent-int-old Feb 6, 2013 10:01:53 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error Feb 6, 2013 10:02:19 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error Feb 6, 2013 10:02:21 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error
(UTC timestamps) While running a 3 matrix jobs with 10 configurations each and 10 slaves (with the web browser open), I get the following utilization PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15209 jenkins 18 0 3366m 690m 18m S 177.2 22.9 203:51.39 java and the web interface is painfully slow. thanks, Andrew On Wed, Feb 6, 2013 at 2:57 PM, Sami Tikka <sjti...@gmail.com> wrote: > Are there any errors/exceptions in Jenkins log files or stdout/stderr? > > -- Sami > > Andrew Melo <andrew.m...@gmail.com> kirjoitti 6.2.2013 kello 22.13: > >> Hey everyone, >> >> Sorry for resurrecting that thread, but are there any solutions for >> this problem? It's basically killing our instance often. >> >> best, >> Andrew >> >> On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo <andrew.m...@gmail.com> wrote: >>> Oh, wow, I didn't notice, but jenkins has autopopulated a user for >>> everyone that ever committed on the project. There's like 30 people, >>> ~4000 commits, so I could see why that would take a while :) >>> >>> On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo <andrew.m...@gmail.com> wrote: >>>> On Wed, Aug 1, 2012 at 11:37 AM, Slide <slide.o....@gmail.com> wrote: >>>>> Can you gist your global config.xml and something from one of your >>>>> jobs as well? Please remember to sanitize it. >>>> >>>> We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ >>>> >>>> And the following is the gist for the job we run each commit (didn't >>>> make it in for some reason...) >>>> >>>> https://gist.github.com/3228599 >>>> >>>>> >>>>> On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo <andrew.m...@gmail.com> wrote: >>>>>> On Wed, Aug 1, 2012 at 11:26 AM, Slide <slide.o....@gmail.com> wrote: >>>>>>> No, because its only looking for the email address because it wants to >>>>>>> send an email to that user. >>>>>> >>>>>> I don't know who's getting emailed. I don't remember setting it up for >>>>>> anything, and we actually wrote some scripts that turn jenkins >>>>>> success/failures into Github issues, so having jenkins also send >>>>>> emails would be redundant. >>>>>> >>>>>> I don't supposed there's a global flag to disable email? (I don't see >>>>>> one at "manage jenkins") >>>>>> >>>>>> -Andrew >>>>>>> >>>>>>> On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <andrew.m...@gmail.com> >>>>>>> wrote: >>>>>>>> On Wed, Aug 1, 2012 at 11:22 AM, Slide <slide.o....@gmail.com> wrote: >>>>>>>>> This is a huge issue with the email-ext plugin as well when it does >>>>>>>>> email address resolution. Quite a number of people have complained >>>>>>>>> about how long it takes. I have yet to come up with a good solution. >>>>>>>>> The perforce plugin has a similar issue. >>>>>>>> >>>>>>>> If I just stick a dummy address in every user's profile, will that >>>>>>>> work? >>>>>>>> >>>>>>>>> >>>>>>>>> slide >>>>>>>>> >>>>>>>>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <vjura...@redhat.com> >>>>>>>>> wrote: >>>>>>>>>> Looks like you it does search for user's email: >>>>>>>>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>>>>>>>> and spends time parsing changelogs: >>>>>>>>>> hudson.scm.SubversionChangeLogParser.parse >>>>>>>>>> >>>>>>>>>> I guess you have quite large instance, otherwise this operation >>>>>>>>>> would be quite >>>>>>>>>> fast. >>>>>>>>>> If you have some job, which has set up option to send an email to >>>>>>>>>> devs who >>>>>>>>>> broke the build, if the user hasn't specified an email, Jenkins >>>>>>>>>> tries to find it >>>>>>>>>> e.g. in git or SVN changelogs and search all projects and builds so >>>>>>>>>> if you >>>>>>>>>> have large instance with several dozen thousands of builds if can >>>>>>>>>> take pretty >>>>>>>>>> long time. >>>>>>>>>> >>>>>>>>>> You can fix it by setting up correct email for the user. >>>>>>>>>> If you have installed git plugin, make sure you have 1.1.16 (I hope >>>>>>>>>> it was >>>>>>>>>> fixed in this version) or higher. Git plugin made this search even >>>>>>>>>> if the user >>>>>>>>>> has set up email correctly >>>>>>>>>> >>>>>>>>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>>>>>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek >>>>>>>>>>> <vjura...@redhat.com> >>>>>>>>>> wrote: >>>>>>>>>>>> On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>>>>>>>>>> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek >>>>>>>>>>>>> <vjura...@redhat.com> >>>>>>>>>> wrote: >>>>>>>>>>>>>> quick way how to look what the thread consuming CPU is doing is >>>>>>>>>>>>>> to do >>>>>>>>>>>>>> thread dump (e.g. using jstack $PID) and use top with threads on >>>>>>>>>>>>>> (H >>>>>>>>>>>>>> option) and then look up, see e.g. >>>>>>>>>>>>>> http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>>>>>>>>>>> consuming-most-cpu/ >>>>>>>>>>>>> >>>>>>>>>>>>> I see. I apparently don't have jstack on this machine :/. Does it >>>>>>>>>>>>> only >>>>>>>>>>>>> come with the JDK, or can I find it somewhere on the JRE? Once I >>>>>>>>>>>>> find >>>>>>>>>>>>> the offending thread, should it be pretty obvious what it does? >>>>>>>>>>>> >>>>>>>>>>>> jstack is part of JDK >>>>>>>>>>>> >>>>>>>>>>>> you can see the stack trace via Jenkins UI navigating to >>>>>>>>>>>> $JENKINS_URL/threadDump but not sure if your (or any) Jenkins >>>>>>>>>>>> version >>>>>>>>>>>> provides thread IDs. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Once you identify the offending thread, it should be obvious what >>>>>>>>>>>> it does >>>>>>>>>>>> (but it may not be obvious why it does what it does:-) >>>>>>>>>>> >>>>>>>>>>> Okay, I installed the jdk, and I looked some more. >>>>>>>>>>> >>>>>>>>>>> Using top, I see one jenkins thread taking the lionsshare of the >>>>>>>>>>> time: >>>>>>>>>>> >>>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>>>>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>>>>>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>>>>>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>>>>>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>>>>>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>>>>>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>>>>>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>>>>>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>>>>>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>>>>>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>>>>>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>>>>>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>>>>>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>>>>>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>>>>>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>>>>>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>>>>>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>>>>>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>>>>>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>>>>>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>>>>>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>>>>>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>>>>>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>>>>>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>>>>>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>>>>>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>>>>>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>>>>>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Then if I run jstack, I get the following backtrace: >>>>>>>>>>> >>>>>>>>>>> https://gist.github.com/3228105 >>>>>>>>>>> >>>>>>>>>>> Does that look useful at all? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Website: http://earl-of-code.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> Andrew Melo >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Website: http://earl-of-code.com >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Andrew Melo >>>>> >>>>> >>>>> >>>>> -- >>>>> Website: http://earl-of-code.com >>>> >>>> >>>> >>>> -- >>>> -- >>>> Andrew Melo >>> >>> >>> >>> -- >>> -- >>> Andrew Melo >> >> >> >> -- >> -- >> Andrew Melo >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-users+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
