On Wednesday, May 7, 2014 1:47:59 AM UTC-7, nigelm wrote: > > > >> >>> >>> From your reply, I am even more concerned with disproportionally high >>> number of the blocked threads (120) compare to offline slaves (2 at the >>> time), as it sounds like it should be closer to 1:1? >>> >> >> Yes, it sounds like there is a race condition between the post disconnect >> tasks and the reconnect tasks: >> https://github.com/jenkinsci/ssh-slaves-plugin/blob/ssh-slaves-1.6/src/main/java/hudson/plugins/sshslaves/SSHLauncher.java#L1152is >> blocking until the slave is connected... but the slave cannot connect >> until the disconnect tasks are complete... >> >> >>> >>> > Do you have 'dead' slaves, and what's your logging configuration like? > > Some slaves appear offline in Jenkins periodically. (The instances are accessible, but Jenkins cannot to connect to it.)
Our logging configuration: winstone - INFO org.apache.sshd - WARNING (default) - INFO I'm tracking down a similar problem, in that our Jenkins instance (which > isn't that large) slows to the state of the UI timing out. > > Not sure if it is the same problem. In our case, the Jenkins server is still responsive, only the SSH slaves are offline. (We did experience a problem with Jenkins 1.562 due to https://issues.jenkins-ci.org/browse/JENKINS-22734. Causing the server to run out of memory and UI to become non-responsive and eventual server hang. Downgrade back to 1.559 seems to solve it. The monitoring plugin helps debug some of these issues.) We also also see occasionally "Proxy Error" when viewing the live build log and viewing the summary, but this is not an SSH issue. java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474) at hudson.remoting.SocketChannelStream$2.write(SocketChannelStream.java:62) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:101) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876) at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1285) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1230) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1426) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1576) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:350) at hudson.remoting.Command.writeTo(Command.java:83) at hudson.remoting.ClassicCommandTransport.write(ClassicCommandTransport.java:51) at hudson.remoting.Channel.send(Channel.java:545) at hudson.remoting.Request.callAsync(Request.java:208) at hudson.remoting.Channel.callAsync(Channel.java:766) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:280) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474) at hudson.remoting.SocketChannelStream$2.write(SocketChannelStream.java:62) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:101) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876) at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1285) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1230) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1426) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1576) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:350) at hudson.remoting.Command.writeTo(Command.java:83) at hudson.remoting.ClassicCommandTransport.write(ClassicCommandTransport.java:51) at hudson.remoting.Channel.send(Channel.java:545) at hudson.remoting.Request.callAsync(Request.java:208) at hudson.remoting.Channel.callAsync(Channel.java:766) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:280) -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
