Thank you. So how does remoting work with respect to end of job notification?
My initial assumption was that it was simply waiting for the forked process to finish, grab the resultant return code, and update the master. Also, any pointers/suggestions as to what information I need/want to get out of the groovy script console? Will certainly look into the queue management code. However, the queue itself is empty (we have more executors than needed at the moment). Jenkins just believes that jobs that actually finished 5 hours ago are still running. - Clark. > So the questions I have are: > > 1. What is the polling cycle on the node monitoring the job and > is it configurable? Not how the remoting works > > 2. Is there a way to get more information out of the node than > just pinging systeminfo on the main Jenkins? Yes via the groovy script console > > 3. Where in the Jenkins code base is the node management code? > Scattered all over, you will want to look into the remoting module, and look at the Slave and Computer classes. But in reality you probably want to look at how the queue works and not node management. You might want to investigate the GC cpu time on the slaves and the master. > > > > > This is the thread dump for one of them > (http://jenkins/node1/systeminfo ) > > Thread Dump > > Channel reader thread: channel > > > > "Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native) > > at java.io.FileInputStream.readBytes(Native Method) > > at > java.io.FileInputStream.read(FileInputStream.java:199) > > at > java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > > at > java.io.BufferedInputStream.read(BufferedInputStream.java:237) > > - locked java.io.BufferedInputStream@2486ae > > at > java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java: > 2249) > > at > java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream. > java:2542) > > at > java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStr > eam.java:2552) > > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297) > > at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > > at > hudson.remoting.Channel$ReaderThread.run(Channel.java:1030) > > > > > > main > > > > "main" Id=1 Group=main WAITING on hudson.remoting.Channel@a17083 > > at java.lang.Object.wait(Native Method) > > - waiting on hudson.remoting.Channel@a17083 > > at java.lang.Object.wait(Object.java:485) > > at hudson.remoting.Channel.join(Channel.java:766) > > at hudson.remoting.Launcher.main(Launcher.java:420) > > at > hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:366) > > at hudson.remoting.Launcher.run(Launcher.java:206) > > at hudson.remoting.Launcher.main(Launcher.java:168) > > > > > > Ping thread for channel hudson.remoting.Channel@a17083:channel > > > > "Ping thread for channel hudson.remoting.Channel@a17083:channel" Id=10 > Group=main TIMED_WAITING > > at java.lang.Thread.sleep(Native Method) > > at hudson.remoting.PingThread.run(PingThread.java:86) > > > > > > pool-1-thread-666 > > > > "pool-1-thread-666" Id=719 Group=main RUNNABLE > > at sun.management.ThreadImpl.dumpThreads0(Native > Method) > > at > sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374) > > at hudson.Functions.getThreadInfos(Functions.java:872) > > at > hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics > .java:93) > > at > hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics > .java:89) > > at > hudson.remoting.UserRequest.perform(UserRequest.java:118) > > at > hudson.remoting.UserRequest.perform(UserRequest.java:48) > > at hudson.remoting.Request$2.run(Request.java:287) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441 > ) > > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at > java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu > tor.java:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > java:908) > > at java.lang.Thread.run(Thread.java:619) > > > > Number of locked synchronizers = 1 > > - > java.util.concurrent.locks.ReentrantLock$NonfairSync@1630de2 > > > > > > Finalizer > > > > "Finalizer" Id=3 Group=system WAITING on > java.lang.ref.ReferenceQueue$Lock@64514 > > at java.lang.Object.wait(Native Method) > > - waiting on java.lang.ref.ReferenceQueue$Lock@64514 > > at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) > > at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) > > at > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > > > > > > Reference Handler > > > > "Reference Handler" Id=2 Group=system WAITING on > java.lang.ref.Reference$Lock@1a12930 > > at java.lang.Object.wait(Native Method) > > - waiting on java.lang.ref.Reference$Lock@1a12930 > > at java.lang.Object.wait(Object.java:485) > > at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > > > > > > Signal Dispatcher > > > > "Signal Dispatcher" Id=4 Group=system RUNNABLE > > > > Thank you, > > > > -Clark. > > The information in this message is for the intended recipient(s) only > and may be the proprietary and/or confidential property of Litle & > Co., LLC, and thus protected from disclosure. If you are not the > intended recipient(s), or an employee or agent responsible for > delivering this message to the intended recipient, you are hereby > notified that any use, dissemination, distribution or copying of this > communication is prohibited. If you have received this communication > in error, please notify Litle & Co. immediately by replying to this > message and then promptly deleting it and your reply permanently from your > computer. The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.
