[JIRA] (JENKINS-49510) Failed connections between JNLP agent and master on Kubernetes

[cesar.tronlo...@gmail.com (JIRA)]

Title: Message Title

Cesar Tron-Lozai created an issue   Jenkins / JENKINS-49510 Failed connections between JNLP agent and master on Kubernetes Issue Type: Bug Assignee: Carlos Sanchez Components: kubernetes-plugin Created: 2018-02-12 16:38 Environment: Jenkins 2.105 Jenkins kubernetes: 1.2 Priority: Major Reporter: Cesar Tron-Lozai I've set up Jenkins to run on a k8s cluster. I'm using the kubernetes plugin to create agent in the cluster. I have a discovery service available at jenkins-discovery.jenkins.svc.cluster.local:50000. I've set up the tunnel property to jenkins-discovery.jenkins.svc.cluster.local:50000 When I start a job, the agent seems to be able to connect but then fails for some unknown reasons.   The agent logs: INFO: Locating server among [http://jenkins-internal.jenkins.svc.cluster.local:8080/] Feb 12, 2018 4:08:37 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins-discovery.jenkins.svc.cluster.local Agent port: 50000 Identity: a1:e6:e0:d2:ad:30:2e:11:47:4f:0f:95:41:d3:d0:a4 Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins-discovery.jenkins.svc.cluster.local:50000 Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Feb 12, 2018 4:08:37 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: a1:e6:e0:d2:ad:30:2e:11:47:4f:0f:95:41:d3:d0:a4 Feb 12, 2018 4:08:39 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Feb 12, 2018 4:08:41 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated It repeats this 4 times and then I see  WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@71b21b06 hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4d5ba314:JNLP4-connect connection to jenkins-discovery.jenkins.svc.cluster.local/100.71.204.63:50000": channel is already closed at hudson.remoting.Channel.send(Channel.java:672) at hudson.remoting.Request$2.run(Request.java:371) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.remoting.Channel$OrderlyShutdown: Command close created at at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1226) at hudson.remoting.Channel$1.handle(Channel.java:561) at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:202) at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:188) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:186) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669) at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283) ... 4 more Caused by: Command close created at at hudson.remoting.Command.<init>(Command.java:62) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1220) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1218) at hudson.remoting.Channel.close(Channel.java:1391) at hudson.remoting.Channel.close(Channel.java:1358) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:745) at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:98) at hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:533) at hudson.remoting.Channel.terminate(Channel.java:1014) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313) at hudson.remoting.Channel.close(Channel.java:1405) at hudson.remoting.Channel.close(Channel.java:1358) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:603) at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168) at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421) at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312) at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418) at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more In the master I see INFO: Created Pod: slave-with-docker-g2k77 in namespace jenkins Feb 12, 2018 4:08:36 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch INFO: Waiting for Pod to be scheduled (0/100): slave-with-docker-g2k77 Feb 12, 2018 4:08:37 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted JNLP4-connect connection #482 from /100.96.10.76:41026 Feb 12, 2018 4:08:38 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted JNLP4-connect connection #483 from /100.96.10.76:41036 Feb 12, 2018 4:08:41 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed WARNING: Computer.threadPoolForRemoting [#553] for slave-with-docker-g2k77 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313) at hudson.remoting.Channel.close(Channel.java:1405) at hudson.remoting.Channel.close(Channel.java:1358) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:603) at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168) at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421) at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312) at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418) at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I can see no meaningful errors in the logs. It really confuses me that it says the connection is successful but then it fails Add Comment   This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.