>From what I see many channels are blocked by classloading, e.g. this one: Classloading might be a long operation, especially when massive classloading saturates your master's network. Just check whether you send big JARs over the network, e.g. AWS SDK (~100Mb) or similar XXXL ones
Computer.threadPoolForRemoting [#484] for obs11-linux-pzz7g1 id=23 Stack Trace is: java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at com.trilead.ssh2.crypto.cipher.CipherOutputStream.internal_write( CipherOutputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherOutputStream.writeBlock( CipherOutputStream.java:101) at com.trilead.ssh2.crypto.cipher.CipherOutputStream.write( CipherOutputStream.java:118) at com.trilead.ssh2.transport.TransportConnection.sendMessage( TransportConnection.java:179) at com.trilead.ssh2.transport.TransportConnection.sendMessage( TransportConnection.java:107) at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager. java:690) - locked <0x00000006d864a0e8> (a java.lang.Object) at com.trilead.ssh2.channel.ChannelManager.sendData(ChannelManager.java:429) - locked <0x00000006d9603858> (a java.lang.Object) at com.trilead.ssh2.channel.ChannelOutputStream.write(ChannelOutputStream. java:63) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90 ) at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) at java.io.OutputStream.write(OutputStream.java:75) at hudson.remoting.ChunkedCommandTransport.writeBlock( ChunkedCommandTransport.java:45) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write( AbstractSynchronousByteArrayCommandTransport.java:46) at hudson.remoting.Channel.send(Channel.java:721) - locked <0x00000006d9602dd8> (a hudson.remoting.Channel) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:143) - locked <0x00000007b7efa6d0> (a hudson.remoting.ProxyOutputStream) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:108) at hudson.remoting.Util.copy(Util.java:56) at hudson.remoting.JarLoaderImpl.writeJarTo(JarLoaderImpl.java:56) at sun.reflect.GeneratedMethodAccessor513.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform( RemoteInvocationHandler.java:929) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call( RemoteInvocationHandler.java:903) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call( RemoteInvocationHandler.java:855) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call( InterceptingExecutorService.java:72) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java: 21) at jenkins.util.ContextResettingExecutorService$2.call( ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call( ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java :1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:624) at java.lang.Thread.run(Thread.java:748) On Monday, February 10, 2020 at 2:16:42 PM UTC+1, Victor Martinez wrote: > > I'm testing the google-compute-engine-plugin and it seems there are a > bunch of blocked threads related to the below hudson.remoting methods: > > > - hudson.remoting.Request.call(Request.java:154) > - hudson.remoting.Channel.send(Channel.java:716) > > See some of the stacktraces: > > Computer.threadPoolForRemoting [#414] > priority:5 - threadId:0x00007f848800c000 - nativeId:0x1c73 - nativeId > (decimal):7283 - state:BLOCKED > stackTrace: > java.lang.Thread.State: BLOCKED (on object monitor) > at hudson.remoting.Request.call(Request.java:154) > - waiting to lock <0x00000007b6b39490> (a hudson.remoting.Channel) > at hudson.remoting.Channel.call(Channel.java:954) > at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:624) > at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:418) > at > com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:318) > at > com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:224) > at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:290) > at hudson.slaves.SlaveComputer$$Lambda$378/348978491.call(Unknown Source) > at > jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) > at > jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > Computer.threadPoolForRemoting [#347] for obs11-linux-r6rpli id=25 > priority:5 - threadId:0x00007f849427b000 - nativeId:0x1a93 - nativeId > (decimal):6803 - state:BLOCKED > stackTrace: > java.lang.Thread.State: BLOCKED (on object monitor) > at hudson.remoting.Channel.send(Channel.java:716) > - waiting to lock <0x00000006d82e4570> (a hudson.remoting.Channel) > at hudson.remoting.Request$2.run(Request.java:382) > at > hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) > at > hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21) > at > jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) > at > jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > *Environment:* > > - Docker Jenkins instance. > - Jenkins Version 2.109 > - Trilead-api 1.05 > - Testing PR: > https://github.com/jenkinsci/google-compute-engine-plugin/pull/183 > - Reported as an issue in the Google Compute Engine plugin: > https://github.com/jenkinsci/google-compute-engine-plugin/pull/184 > > > I created a few dumps and uploaded them to https://fastthread.io/: > > - > > https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDIvMTAvLS1vdXRwdXQtMi5sb2ctLTEyLTEwLTU5OzstLW91dHB1dC0xLmxvZy0tMTItMTAtNTk7Oy0tb3V0cHV0LmxvZy0tMTItMTAtNTk= > > > *Questions:* > > - Have you seen that particular behaviour? > - Any clues how I can debug it further? > - Is that an expected behaviour in the remoting itself? Or a kind of > bug? > > > Thanks > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/161b7b92-ac39-4744-9f85-6792dd23f83b%40googlegroups.com.
