On 9 May 2012 16:31, Wright, Clark <cwri...@litle.com> wrote: > Thank you. > > So how does remoting work with respect to end of job notification? > > My initial assumption was that it was simply waiting for the forked process > to finish, grab the resultant return code, and update the master. >
well you could look at it like that, in actuality the better way to look at it is as more or a distributed jvm. The master sends a closure to the slave, the closure forks the child process and when the child process completes the closure should return the result to the master. > Also, any pointers/suggestions as to what information I need/want to get out > of the groovy script console? > > Will certainly look into the queue management code. However, the queue > itself is empty (we have more executors than needed at the moment). Jenkins > just believes that jobs that actually finished 5 hours ago are still running. Smells like a GC issue but I could be wrong. > > - Clark. > >> So the questions I have are: >> >> 1. What is the polling cycle on the node monitoring the job and >> is it configurable? > > Not how the remoting works > >> >> 2. Is there a way to get more information out of the node than >> just pinging systeminfo on the main Jenkins? > > Yes via the groovy script console > >> >> 3. Where in the Jenkins code base is the node management code? >> > > Scattered all over, you will want to look into the remoting module, and look > at the Slave and Computer classes. > > But in reality you probably want to look at how the queue works and not node > management. > > You might want to investigate the GC cpu time on the slaves and the master. > >> >> >> >> >> This is the thread dump for one of them >> (http://jenkins/node1/systeminfo ) >> >> Thread Dump >> >> Channel reader thread: channel >> >> >> >> "Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native) >> >> at java.io.FileInputStream.readBytes(Native Method) >> >> at >> java.io.FileInputStream.read(FileInputStream.java:199) >> >> at >> java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >> >> at >> java.io.BufferedInputStream.read(BufferedInputStream.java:237) >> >> - locked java.io.BufferedInputStream@2486ae >> >> at >> java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java: >> 2249) >> >> at >> java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream. >> java:2542) >> >> at >> java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStr >> eam.java:2552) >> >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) >> >> at >> java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) >> >> at >> hudson.remoting.Channel$ReaderThread.run(Channel.java:1030) >> >> >> >> >> >> main >> >> >> >> "main" Id=1 Group=main WAITING on hudson.remoting.Channel@a17083 >> >> at java.lang.Object.wait(Native Method) >> >> - waiting on hudson.remoting.Channel@a17083 >> >> at java.lang.Object.wait(Object.java:485) >> >> at hudson.remoting.Channel.join(Channel.java:766) >> >> at hudson.remoting.Launcher.main(Launcher.java:420) >> >> at >> hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366) >> >> at hudson.remoting.Launcher.run(Launcher.java:206) >> >> at hudson.remoting.Launcher.main(Launcher.java:168) >> >> >> >> >> >> Ping thread for channel hudson.remoting.Channel@a17083:channel >> >> >> >> "Ping thread for channel hudson.remoting.Channel@a17083:channel" Id=10 >> Group=main TIMED_WAITING >> >> at java.lang.Thread.sleep(Native Method) >> >> at hudson.remoting.PingThread.run(PingThread.java:86) >> >> >> >> >> >> pool-1-thread-666 >> >> >> >> "pool-1-thread-666" Id=719 Group=main RUNNABLE >> >> at sun.management.ThreadImpl.dumpThreads0(Native >> Method) >> >> at >> sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374) >> >> at hudson.Functions.getThreadInfos(Functions.java:872) >> >> at >> hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics >> .java:93) >> >> at >> hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics >> .java:89) >> >> at >> hudson.remoting.UserRequest.perform(UserRequest.java:118) >> >> at >> hudson.remoting.UserRequest.perform(UserRequest.java:48) >> >> at hudson.remoting.Request$2.run(Request.java:287) >> >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441 >> ) >> >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> >> at >> java.util.concurrent.FutureTask.run(FutureTask.java:138) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu >> tor.java:886) >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. >> java:908) >> >> at java.lang.Thread.run(Thread.java:619) >> >> >> >> Number of locked synchronizers = 1 >> >> - >> java.util.concurrent.locks.ReentrantLock$NonfairSync@1630de2 >> >> >> >> >> >> Finalizer >> >> >> >> "Finalizer" Id=3 Group=system WAITING on >> java.lang.ref.ReferenceQueue$Lock@64514 >> >> at java.lang.Object.wait(Native Method) >> >> - waiting on java.lang.ref.ReferenceQueue$Lock@64514 >> >> at >> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) >> >> at >> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) >> >> at >> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >> >> >> >> >> >> Reference Handler >> >> >> >> "Reference Handler" Id=2 Group=system WAITING on >> java.lang.ref.Reference$Lock@1a12930 >> >> at java.lang.Object.wait(Native Method) >> >> - waiting on java.lang.ref.Reference$Lock@1a12930 >> >> at java.lang.Object.wait(Object.java:485) >> >> at >> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >> >> >> >> >> >> Signal Dispatcher >> >> >> >> "Signal Dispatcher" Id=4 Group=system RUNNABLE >> >> >> >> Thank you, >> >> >> >> -Clark. >> >> The information in this message is for the intended recipient(s) only >> and may be the proprietary and/or confidential property of Litle & >> Co., LLC, and thus protected from disclosure. If you are not the >> intended recipient(s), or an employee or agent responsible for >> delivering this message to the intended recipient, you are hereby >> notified that any use, dissemination, distribution or copying of this >> communication is prohibited. If you have received this communication >> in error, please notify Litle & Co. immediately by replying to this >> message and then promptly deleting it and your reply permanently from your >> computer. > > The information in this message is for the intended recipient(s) only and may > be the proprietary and/or confidential property of Litle & Co., LLC, and thus > protected from disclosure. If you are not the intended recipient(s), or an > employee or agent responsible for delivering this message to the intended > recipient, you are hereby notified that any use, dissemination, distribution > or copying of this communication is prohibited. If you have received this > communication in error, please notify Litle & Co. immediately by replying to > this message and then promptly deleting it and your reply permanently from > your computer.
