I agree with Ivan. There are differences, sometimes subtle, between how
different agents behave for various reasons. The basic operations of the
plugins and the protocol should be the same. There are can be
differences in resource usage, platforms, etc. Sometimes plugins will
behave differently on different one.
Try out some of Ivan's suggests or other troubleshooting like that and
figure out how to isolate the problem.
Jeff Thompson
On 2/2/21 10:20 AM, Ivan Fernandez Calvo wrote:
>I have 2 different kind of jenkins agents.
>1. java web start
>2. ssh
>If the memory was issue then wouldn't it fail on both cases?
Not necessarily, starting by the point that are different ways to
establish the connection, the JNLP agents could not update the
remoting jar file (depends on your configurations), so you can be
running different versions of remoting. I agree with Jeff looks like
an OOM issue, review my comments
at https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ
my recommendation is to fix the mem for the remoting process to 1024M
(-Xmx1024m -Xms1024m) see if the issue disappear or change, if
disappear, you would have to adjust the remoting process memory to the
right one between 256M-1024M, to use 512M usually is safe a not too
much (but depends on your agents' memory we do not know how much they
have)
El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1,
jigarra...@gmail.com escribió:
On Monday, January 4, 2021 at 12:37:52 PM UTC-5
jtho...@cloudbees.com wrote:
When I saw something like this in the past, it was because the
process was running out of resources. Specifically when
running SpotBugs, I got out-of-memory errors. I had to modify
the pom to allocate more memory.
I have 2 different kind of jenkins agents.
1. java web start
2. ssh
If the memory was issue then wouldn't it fail on both cases?
I do see "Agent went offline during build Connection was broker:
java.io.StreamCorruptedException: invalid stream header:".
I recommend ensuring your build runs normally on the agent
without the additional complexities of the Jenkins
environment. This might show where additional resources are
needed. If that all passes, then continue on to the additional
troubleshooting steps involving the Jenkins controller and agent.
Jeff Thompson
On 12/30/20 3:40 PM, jiga...@gmail.com wrote:
Interestingly, whenever Jenkins agent is executing
PMD/Findbugs task, it runs into " Resource temporarily
unavailable".
I have
* verified TCP parameters
i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes
& tcp_fin_timeout ; All of them have higher values then
what cloudbees recommended at
https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected
<https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected>
* set heap space to 2-4Gb
* disabled ping thread from Jenkins agent by setting
"-Dhudson.remoting.Launcher.pingIntervalSec=-1"
Please let me know if there is anything else I should consider.
On Friday, December 11, 2020 at 12:11:39 PM UTC-5
kuisat...@gmail.com wrote:
The most common issue is related to disconnections
because there is no traffic between the Jenkins instance
and the agent, for that, you have to tune the TCP stack
of your OS (see
https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected
<https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected>),
or enable the keepalive option in the SSH protocol this
can be configured by setting ClientAliveInterval or
TCPKeepAlive on the SSH server (/etc/ssh/sshd_config),
also by setting ServerAliveInterval or TCPKeepAlive
options for the user connection (/etc/ssh/ssh_config or
~/.ssh/ssh_config)
https://www.freebsd.org/cgi/man.cgi?sshd_config(5
<https://www.freebsd.org/cgi/man.cgi?sshd_config(5>)
https://www.freebsd.org/cgi/man.cgi?ssh_config(5
<https://www.freebsd.org/cgi/man.cgi?ssh_config(5>)
Also, check you follow the best practices to configure
your SSH agents and enable SSH verbose log output in your
service (see
https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md
<https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md>)
El miércoles, 9 de diciembre de 2020 a las 17:53:19
UTC+1, jiga...@gmail.com escribió:
Hello Jenkins community,
I have setup Jenkins agents over SSH and this agent
frequently goes offline with Resource temporarily
unavailable error. I had to configure my Jenkins
agents to communicate over JNLP. Any advise on how to
fix this issue?
Jenkins v2.249.1
[12/04/20 15:23:00] [SSH] Checking java version of java
[12/04/20 15:23:01] [SSH] java -version returned
1.8.0_202.
[12/04/20 15:23:01] [SSH] Starting sftp client.
[12/04/20 15:23:03] [SSH] Remote file system root
$JENKINS_SSH_DATA does not exist. Will try to create
it...
[12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
[12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
Expanded the channel window size to 4MB
[12/04/20 15:23:03] [SSH] Starting agent process: cd
"$JENKINS_SSH_DATA" && java -jar remoting.jar
-workDir $JENKINS_SSH_DATA -jar-cache
$JENKINS_SSH_DATA/remoting/jarCache
Dec 04, 2020 3:23:29 PM
org.jenkinsci.remoting.engine.WorkDirManager
initializeWorkDir
INFO: Using $JENKINS_SSH_DATA/remoting as a remoting
work directory
Dec 04, 2020 3:23:29 PM
org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to
$JENKINS_SSH_DATA/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.5
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
The Agent is connected, disconnect it before to try
to connect it again.
Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS
discoverGitSystemConfig
WARNING: Exception caught during execution of command
'[git, config, --system, --edit]' in '$GIT_PATH/bin',
return code '128', error message 'fatal: Invalid path
'$GIT_PATH/etc': No such file or directory
'
Dec 04, 2020 3:33:58 PM
org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
WARNING: locking
FileBasedConfig[$JENKINS_PATH/.config/jgit/config]
failed after 5 retries
Dec 04, 2020 3:33:59 PM
org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class
com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1;
see:
https://jenkins.io/redirect/serialization-of-anonymous-classes/
<https://jenkins.io/redirect/serialization-of-anonymous-classes/>
Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request
hudson.remoting.Request$2@493c5a4e
java.io.IOException: Resource temporarily unavailable
at java.io.FileOutputStream.writeBytes(Native Method)
at
java.io.FileOutputStream.write(FileOutputStream.java:313)
at
hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
at
hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
at
hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
at
hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
at hudson.remoting.Channel.send(Channel.java:766)
at hudson.remoting.Request$2.run(Request.java:388)
at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
WARNING: Failed to send back a reply to the request
hudson.remoting.Request$2@2511e2d4
java.io.IOException: Resource temporarily unavailable
at java.io.FileOutputStream.writeBytes(Native Method)
at
java.io.FileOutputStream.write(FileOutputStream.java:326)
at
hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
at
hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
at
hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
at
hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
at java.io.OutputStream.write(OutputStream.java:75)
at
hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
at hudson.remoting.Channel.send(Channel.java:766)
at hudson.remoting.Request$2.run(Request.java:388)
at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR: Connection terminated
java.io.StreamCorruptedException: invalid stream
header: 00025B42
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
at
java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
at
hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1
onDead
INFO: Ping failed. Terminating the channel channel.
java.util.concurrent.TimeoutException: Ping started
at 1607120910589 hasn't completed by 1607121150590
at
hudson.remoting.PingThread.ping(PingThread.java:134)
at hudson.remoting.PingThread.run(PingThread.java:90)
Thanks,
Jigar R
--
You received this message because you are subscribed to the
Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to jenkinsci-users+unsubscr...@googlegroups.com
<mailto:jenkinsci-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/099267e4-36f2-4b1e-adf1-4ed6f6d5bf90n%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-users/099267e4-36f2-4b1e-adf1-4ed6f6d5bf90n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups "Jenkins
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/2999d743-ff8c-aa70-fcd5-e2ed64bcb854%40cloudbees.com.
