[JIRA] (JENKINS-60673) Swarm agents sometimes disconnect and stay that way forever

[jimkli...@gmail.com (JIRA)]

Title: Message Title

Jim Klimov created an issue   Jenkins / JENKINS-60673 Swarm agents sometimes disconnect and stay that way forever Issue Type: Improvement Assignee: Unassigned Components: swarm-plugin Created: 2020-01-07 09:11 Priority: Major Reporter: Jim Klimov For context, our Jenkins master server JVM is dedicated to dispatching, with some of the infra jobs handled by an agent running on the same machine in a different account. Some time ago this agent was remade from SSH to Swarm, and over some uptime it tends to disconnect, and or not-connect when the master is too busy We had an outage starting a few days ago that was only noticed now as we came back to work, e.g. this being the last logged line: Jan 05 08:59:37 jenkins2 bash[15117]: INFO: Failed to send back a reply to the request hudson.remoting.Request$2@525facc1: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write The agent JVM continued running, so neither some logic inside the agent, nor systemd, would restart it to actually reconnect and keep the real service provided. Here the JVM seems to be alive, so nothing is restarted by the OS, and the agent just disappears from master since there is no connection. In fact, preceding lines point to insufficient memory (and I have no idea how much we should throw at it, because with whatever settings we tried it works for days and weeks and then suddenly it does not). Jan 05 08:59:37 jenkins2 bash[15117]: at java.lang.Thread.run(Thread.java:748) Jan 05 08:59:37 jenkins2 bash[15117]: Caused by: java.lang.OutOfMemoryError: Java heap space Add Comment   This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.203916.1578388273000.3494.1578388320180%40Atlassian.JIRA.