On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [email protected] wrote:
> >I have 2 different kind of jenkins agents. > >1. java web start > >2. ssh > >If the memory was issue then wouldn't it fail on both cases? > > Not necessarily, starting by the point that are different ways to > establish the connection, the JNLP agents could not update the remoting jar > file (depends on your configurations), so you can be running different > versions of remoting. I agree with Jeff looks like an OOM issue, review my > comments at > https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ > my recommendation is to fix the mem for the remoting process to 1024M > (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear, > you would have to adjust the remoting process memory to the right one > between 256M-1024M, to use 512M usually is safe a not too much (but depends > on your agents' memory we do not know how much they have) > I created SSH agent with -Xmx1024m -Xms1024m. I do see bunch of Warnings about "Ping failed. Terminating the channel channel". I got following error: " WARNING: Failed to send back a reply to te rquest hudson.remoting.Request$.... java.io.IOException: Resource temporarily unavailable at java.io.FileOutputStream.writeBytes(Native method) ..... " > > El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1, [email protected] > escribió: > >> On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [email protected] >> wrote: >> >>> When I saw something like this in the past, it was because the process >>> was running out of resources. Specifically when running SpotBugs, I got >>> out-of-memory errors. I had to modify the pom to allocate more memory. >>> >> I have 2 different kind of jenkins agents. >> 1. java web start >> 2. ssh >> If the memory was issue then wouldn't it fail on both cases? >> >> I do see "Agent went offline during build Connection was broker: >> java.io.StreamCorruptedException: invalid stream header:". >> >>> I recommend ensuring your build runs normally on the agent without the >>> additional complexities of the Jenkins environment. This might show where >>> additional resources are needed. If that all passes, then continue on to >>> the additional troubleshooting steps involving the Jenkins controller and >>> agent. >>> >>> Jeff Thompson >>> On 12/30/20 3:40 PM, [email protected] wrote: >>> >>> Interestingly, whenever Jenkins agent is executing PMD/Findbugs task, it >>> runs into " Resource temporarily unavailable". >>> I have >>> >>> - verified TCP parameters >>> i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes >>> & tcp_fin_timeout ; All of them have higher values then what cloudbees >>> recommended at >>> >>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected >>> >>> - set heap space to 2-4Gb >>> - disabled ping thread from Jenkins agent by setting >>> "-Dhudson.remoting.Launcher.pingIntervalSec=-1" >>> >>> Please let me know if there is anything else I should consider. >>> On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [email protected] >>> wrote: >>> >>>> The most common issue is related to disconnections because there is no >>>> traffic between the Jenkins instance and the agent, for that, you have to >>>> tune the TCP stack of your OS (see >>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected), >>>> >>>> or enable the keepalive option in the SSH protocol this can be configured >>>> by setting ClientAliveInterval or TCPKeepAlive on the SSH server >>>> (/etc/ssh/sshd_config), also by setting ServerAliveInterval or >>>> TCPKeepAlive >>>> options for the user connection (/etc/ssh/ssh_config or ~/.ssh/ssh_config) >>>> >>>> https://www.freebsd.org/cgi/man.cgi?sshd_config(5) >>>> https://www.freebsd.org/cgi/man.cgi?ssh_config(5) >>>> >>>> Also, check you follow the best practices to configure your SSH agents >>>> and enable SSH verbose log output in your service (see >>>> https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md >>>> ) >>>> >>>> El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1, >>>> [email protected] escribió: >>>> >>>>> Hello Jenkins community, >>>>> >>>>> I have setup Jenkins agents over SSH and this agent frequently goes >>>>> offline with Resource temporarily unavailable error. I had to configure >>>>> my >>>>> Jenkins agents to communicate over JNLP. Any advise on how to fix this >>>>> issue? >>>>> >>>>> Jenkins v2.249.1 >>>>> >>>>> [12/04/20 15:23:00] [SSH] Checking java version of java >>>>> [12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202. >>>>> [12/04/20 15:23:01] [SSH] Starting sftp client. >>>>> [12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA >>>>> does not exist. Will try to create it... >>>>> [12/04/20 15:23:03] [SSH] Copying latest remoting.jar... >>>>> [12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes. >>>>> Expanded the channel window size to 4MB >>>>> [12/04/20 15:23:03] [SSH] Starting agent process: cd >>>>> "$JENKINS_SSH_DATA" && java -jar remoting.jar -workDir $JENKINS_SSH_DATA >>>>> -jar-cache $JENKINS_SSH_DATA/remoting/jarCache >>>>> Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager >>>>> initializeWorkDir >>>>> INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory >>>>> Dec 04, 2020 3:23:29 PM org.jenkinsci.remoting.engine.WorkDirManager >>>>> setupLogging >>>>> INFO: Both error and output logs will be printed to >>>>> $JENKINS_SSH_DATA/remoting >>>>> <===[JENKINS REMOTING CAPACITY]===>channel started >>>>> Remoting version: 4.5 >>>>> This is a Unix agent >>>>> Evacuated stdout >>>>> Agent successfully connected and online >>>>> The Agent is connected, disconnect it before to try to connect it >>>>> again. >>>>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS >>>>> discoverGitSystemConfig >>>>> WARNING: Exception caught during execution of command '[git, config, >>>>> --system, --edit]' in '$GIT_PATH/bin', return code '128', error message >>>>> 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory >>>>> ' >>>>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS$FileStoreAttributes >>>>> saveToConfig >>>>> WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config] >>>>> failed after 5 retries >>>>> Dec 04, 2020 3:33:59 PM >>>>> org.jenkinsci.remoting.util.AnonymousClassWarnings warn >>>>> WARNING: Attempt to (de-)serialize anonymous class >>>>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1; >>>>> >>>>> see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ >>>>> Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run >>>>> WARNING: Failed to send back a reply to the request >>>>> hudson.remoting.Request$2@493c5a4e >>>>> java.io.IOException: Resource temporarily unavailable >>>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>>> at java.io.FileOutputStream.write(FileOutputStream.java:313) >>>>> at >>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62) >>>>> at >>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46) >>>>> at >>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>>>> at hudson.remoting.Channel.send(Channel.java:766) >>>>> at hudson.remoting.Request$2.run(Request.java:388) >>>>> at >>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> >>>>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run >>>>> WARNING: Failed to send back a reply to the request >>>>> hudson.remoting.Request$2@2511e2d4 >>>>> java.io.IOException: Resource temporarily unavailable >>>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326) >>>>> at >>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85) >>>>> at >>>>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54) >>>>> at java.io.OutputStream.write(OutputStream.java:75) >>>>> at >>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) >>>>> at >>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46) >>>>> at hudson.remoting.Channel.send(Channel.java:766) >>>>> at hudson.remoting.Request$2.run(Request.java:388) >>>>> at >>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> >>>>> ERROR: Connection terminated >>>>> java.io.StreamCorruptedException: invalid stream header: 00025B42 >>>>> at >>>>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866) >>>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) >>>>> at >>>>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) >>>>> at hudson.remoting.Command.readFrom(Command.java:142) >>>>> at hudson.remoting.Command.readFrom(Command.java:128) >>>>> at >>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) >>>>> at >>>>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) >>>>> Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead >>>>> INFO: Ping failed. Terminating the channel channel. >>>>> java.util.concurrent.TimeoutException: Ping started at 1607120910589 >>>>> hasn't completed by 1607121150590 >>>>> at hudson.remoting.PingThread.ping(PingThread.java:134) >>>>> at hudson.remoting.PingThread.run(PingThread.java:90) >>>>> >>>>> Thanks, >>>>> Jigar R >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com.
