| Jeff Thompson Hello, we also caught this issue. and I think I can bring some explanation ( not fully ) , this time it is related to Slave startup routines Agent went offline with after several seconds :
XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m -Dgroovy.use.classvalue=true -jar remoting.jar -workDir /data/pentaho/jenkins/test-head01
Jan 24, 2019 3:16:49 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /data/xxxxxxxxxxxxxxxx/remoting as a remoting work directory
Both error and output logs will be printed to /data/xxxxxxxxxxxxxxx/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.27
This is a Unix agent
Evacuated stdout
just before slave ph-slave-01 gets online ...
executing prepare script ...
setting up slave ph-slave-01 ...
slave setup done.
Jan 24, 2019 3:16:51 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https:ERROR: null
java.util.concurrent.CancellationException
at java.util.concurrent.FutureTask.report(FutureTask.java:121)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:902)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[01/24/19 03:17:08] Launch failed - cleaning up connection
[01/24/19 03:17:08] [SSH] Connection closed.
ERROR: Connection terminated
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
I've found out that some jobs even managed to start, but then abruptly were terminated. I found in master log some possible explanation: mail watcher could not send email about slave started, some mailer configuration was changed recently, ( and after that change slave could not survive start).
Jan 24, 2019 3:16:45 AM org.jenkinsci.plugins.mailwatcher.MailWatcherNotification log
INFO: mail-watcher-plugin: unable to notify
javax.mail.MessagingException: Could not connect to SMTP host: xxxxxxxxx, port: 465;
nested exception is:
java.net.SocketTimeoutException: connect timed out
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1934)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:638)
at javax.mail.Service.connect(Service.java:295)
at javax.mail.Service.connect(Service.java:176)
at javax.mail.Service.connect(Service.java:125)
at javax.mail.Transport.send0(Transport.java:194)
at javax.mail.Transport.send(Transport.java:124)
at org.jenkinsci.plugins.mailwatcher.MailWatcherMailer.send(MailWatcherMailer.java:135)
at org.jenkinsci.plugins.mailwatcher.MailWatcherMailer.send(MailWatcherMailer.java:128)
at org.jenkinsci.plugins.mailwatcher.MailWatcherNotification.send(MailWatcherNotification.java:156)
at org.jenkinsci.plugins.mailwatcher.WatcherComputerListener$Notification$Builder.send(WatcherComputerListener.java:181)
at org.jenkinsci.plugins.mailwatcher.WatcherComputerListener.onOnline(WatcherComputerListener.java:101)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:693)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:432)
at hudson.plugins.sshslaves.SSHLauncher.startAgent(SSHLauncher.java:1034)
at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:128)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:868)
at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:833)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
so Jenkins could sent notification mail. Mailer listened to both : 25 and 465, after changes only to 25/tcp port. When I fix that configuration on master for mailer, all errors went off, and slave can start.
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.27
This is a Unix agent
Evacuated stdout
just before slave ph-slave-01 gets online ...
executing prepare script ...
setting up slave ph-slave-01 ...
slave setup done.
Jan 24, 2019 3:19:44 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https:[StartupTrigger] - Scanning jobs for node ph-slave-01
Agent successfully connected and online
you can note : Agent successfully connected and online I hope it means that until master could not notify/just go clean with startup procedure for Slave start event, it breaks connection with it. |
