I see some serialization fails and this breaks the channel, the plugin that
causes the exception seems https://github.com/jenkinsci/tasks-plugin, and
the `[Deprecated] Scan workspace for open tasks` I think matters, this
plugins has been integrated into
https://github.com/jenkinsci/warnings-ng-plugin and
https://github.com/jenkinsci/analysis-model
ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due
to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
at
hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
at
hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
at
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1919)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:428)
El jue, 4 feb 2021 a las 14:58, Jigar R (<jigarrathod...@gmail.com>)
escribió:
> Hello Ivan,
>
> Attached the logs in the email
>
> - jenkins.log - jenkins build output
> - Jenkins-agent.log - output from jenkins ssh agent
>
> Jenkins SSH agent was created with following:
>
> -
> - launch method : launch agents via ssh
> - JavaPath: $JAVA_HOME
> - JVM options: -Xmx2048m -Xms2048m
> - Use TCP_NODELAY flag on the ssh connection - enabled
>
> Environment information
>
>
> - Jenkins v2.249.1
> - RH6
> - SSH agent plugin 1.20
> - SSH slaves plugin 1.30.4
>
>
> On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 kuisat...@gmail.com
> wrote:
>
>> Weird, Could you share a screencapture of what you configurate? Also the
>> whole exception those lines alone mean nothing. Know the version of Jenkins
>> and the version of the SSH build agents plugins you use can help, the
>> memory you have in your agents, if they are bare metal or cloud. In overall
>> if you want help please provide more context.
>>
>> El mar, 2 feb 2021 a las 22:52, Jigar R (<jigarra...@gmail.com>)
>> escribió:
>>
>>>
>>>
>>> On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 kuisat...@gmail.com
>>> wrote:
>>>
>>>> >I have 2 different kind of jenkins agents.
>>>> >1. java web start
>>>> >2. ssh
>>>> >If the memory was issue then wouldn't it fail on both cases?
>>>>
>>>> Not necessarily, starting by the point that are different ways to
>>>> establish the connection, the JNLP agents could not update the remoting jar
>>>> file (depends on your configurations), so you can be running different
>>>> versions of remoting. I agree with Jeff looks like an OOM issue, review my
>>>> comments at
>>>> https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ
>>>> my recommendation is to fix the mem for the remoting process to 1024M
>>>> (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear,
>>>> you would have to adjust the remoting process memory to the right one
>>>> between 256M-1024M, to use 512M usually is safe a not too much (but depends
>>>> on your agents' memory we do not know how much they have)
>>>>
>>>
>>> I created SSH agent with -Xmx1024m -Xms1024m.
>>> I do see bunch of Warnings about "Ping failed. Terminating the channel
>>> channel".
>>> I got following error:
>>> "
>>> WARNING: Failed to send back a reply to te rquest
>>> hudson.remoting.Request$....
>>> java.io.IOException: Resource temporarily unavailable
>>> at java.io.FileOutputStream.writeBytes(Native method)
>>> .....
>>> "
>>>
>>>
>>>>
>>>> El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1,
>>>> jigarra...@gmail.com escribió:
>>>>
>>>>> On Monday, January 4, 2021 at 12:37:52 PM UTC-5 jtho...@cloudbees.com
>>>>> wrote:
>>>>>
>>>>>> When I saw something like this in the past, it was because the
>>>>>> process was running out of resources. Specifically when running
>>>>>> SpotBugs, I
>>>>>> got out-of-memory errors. I had to modify the pom to allocate more
>>>>>> memory.
>>>>>>
>>>>> I have 2 different kind of jenkins agents.
>>>>> 1. java web start
>>>>> 2. ssh
>>>>> If the memory was issue then wouldn't it fail on both cases?
>>>>>
>>>>> I do see "Agent went offline during build Connection was broker:
>>>>> java.io.StreamCorruptedException: invalid stream header:".
>>>>>
>>>>>> I recommend ensuring your build runs normally on the agent without
>>>>>> the additional complexities of the Jenkins environment. This might show
>>>>>> where additional resources are needed. If that all passes, then continue
>>>>>> on
>>>>>> to the additional troubleshooting steps involving the Jenkins controller
>>>>>> and agent.
>>>>>>
>>>>>> Jeff Thompson
>>>>>> On 12/30/20 3:40 PM, jiga...@gmail.com wrote:
>>>>>>
>>>>>> Interestingly, whenever Jenkins agent is executing PMD/Findbugs task,
>>>>>> it runs into " Resource temporarily unavailable".
>>>>>> I have
>>>>>>
>>>>>> - verified TCP parameters
>>>>>> i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes
>>>>>> & tcp_fin_timeout ; All of them have higher values then what cloudbees
>>>>>> recommended at
>>>>>>
>>>>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected
>>>>>> - set heap space to 2-4Gb
>>>>>> - disabled ping thread from Jenkins agent by setting
>>>>>> "-Dhudson.remoting.Launcher.pingIntervalSec=-1"
>>>>>>
>>>>>> Please let me know if there is anything else I should consider.
>>>>>> On Friday, December 11, 2020 at 12:11:39 PM UTC-5 kuisat...@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> The most common issue is related to disconnections because there is
>>>>>>> no traffic between the Jenkins instance and the agent, for that, you
>>>>>>> have
>>>>>>> to tune the TCP stack of your OS (see
>>>>>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected),
>>>>>>> or enable the keepalive option in the SSH protocol this can be
>>>>>>> configured
>>>>>>> by setting ClientAliveInterval or TCPKeepAlive on the SSH server
>>>>>>> (/etc/ssh/sshd_config), also by setting ServerAliveInterval or
>>>>>>> TCPKeepAlive
>>>>>>> options for the user connection (/etc/ssh/ssh_config or
>>>>>>> ~/.ssh/ssh_config)
>>>>>>>
>>>>>>> https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
>>>>>>> https://www.freebsd.org/cgi/man.cgi?ssh_config(5)
>>>>>>>
>>>>>>> Also, check you follow the best practices to configure your SSH
>>>>>>> agents and enable SSH verbose log output in your service (see
>>>>>>> https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md
>>>>>>> )
>>>>>>>
>>>>>>> El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1,
>>>>>>> jiga...@gmail.com escribió:
>>>>>>>
>>>>>>>> Hello Jenkins community,
>>>>>>>>
>>>>>>>> I have setup Jenkins agents over SSH and this agent frequently goes
>>>>>>>> offline with Resource temporarily unavailable error. I had to
>>>>>>>> configure my
>>>>>>>> Jenkins agents to communicate over JNLP. Any advise on how to fix this
>>>>>>>> issue?
>>>>>>>>
>>>>>>>> Jenkins v2.249.1
>>>>>>>>
>>>>>>>> [12/04/20 15:23:00] [SSH] Checking java version of java
>>>>>>>> [12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
>>>>>>>> [12/04/20 15:23:01] [SSH] Starting sftp client.
>>>>>>>> [12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA
>>>>>>>> does not exist. Will try to create it...
>>>>>>>> [12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
>>>>>>>> [12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
>>>>>>>> Expanded the channel window size to 4MB
>>>>>>>> [12/04/20 15:23:03] [SSH] Starting agent process: cd
>>>>>>>> "$JENKINS_SSH_DATA" && java -jar remoting.jar -workDir
>>>>>>>> $JENKINS_SSH_DATA
>>>>>>>> -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
>>>>>>>> Dec 04, 2020 3:23:29 PM
>>>>>>>> org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
>>>>>>>> INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
>>>>>>>> Dec 04, 2020 3:23:29 PM
>>>>>>>> org.jenkinsci.remoting.engine.WorkDirManager setupLogging
>>>>>>>> INFO: Both error and output logs will be printed to
>>>>>>>> $JENKINS_SSH_DATA/remoting
>>>>>>>> <===[JENKINS REMOTING CAPACITY]===>channel started
>>>>>>>> Remoting version: 4.5
>>>>>>>> This is a Unix agent
>>>>>>>> Evacuated stdout
>>>>>>>> Agent successfully connected and online
>>>>>>>> The Agent is connected, disconnect it before to try to connect it
>>>>>>>> again.
>>>>>>>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS
>>>>>>>> discoverGitSystemConfig
>>>>>>>> WARNING: Exception caught during execution of command '[git,
>>>>>>>> config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error
>>>>>>>> message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
>>>>>>>> '
>>>>>>>> Dec 04, 2020 3:33:58 PM
>>>>>>>> org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
>>>>>>>> WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config]
>>>>>>>> failed after 5 retries
>>>>>>>> Dec 04, 2020 3:33:59 PM
>>>>>>>> org.jenkinsci.remoting.util.AnonymousClassWarnings warn
>>>>>>>> WARNING: Attempt to (de-)serialize anonymous class
>>>>>>>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1;
>>>>>>>> see:
>>>>>>>> https://jenkins.io/redirect/serialization-of-anonymous-classes/
>>>>>>>> Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
>>>>>>>> WARNING: Failed to send back a reply to the request
>>>>>>>> hudson.remoting.Request$2@493c5a4e
>>>>>>>> java.io.IOException: Resource temporarily unavailable
>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:313)
>>>>>>>> at
>>>>>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
>>>>>>>> at
>>>>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
>>>>>>>> at hudson.remoting.Channel.send(Channel.java:766)
>>>>>>>> at hudson.remoting.Request$2.run(Request.java:388)
>>>>>>>> at
>>>>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>
>>>>>>>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
>>>>>>>> WARNING: Failed to send back a reply to the request
>>>>>>>> hudson.remoting.Request$2@2511e2d4
>>>>>>>> java.io.IOException: Resource temporarily unavailable
>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>>>>>>> at
>>>>>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
>>>>>>>> at java.io.OutputStream.write(OutputStream.java:75)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
>>>>>>>> at
>>>>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
>>>>>>>> at hudson.remoting.Channel.send(Channel.java:766)
>>>>>>>> at hudson.remoting.Request$2.run(Request.java:388)
>>>>>>>> at
>>>>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>
>>>>>>>> ERROR: Connection terminated
>>>>>>>> java.io.StreamCorruptedException: invalid stream header: 00025B42
>>>>>>>> at
>>>>>>>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
>>>>>>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
>>>>>>>> at
>>>>>>>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
>>>>>>>> at hudson.remoting.Command.readFrom(Command.java:142)
>>>>>>>> at hudson.remoting.Command.readFrom(Command.java:128)
>>>>>>>> at
>>>>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
>>>>>>>> at
>>>>>>>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
>>>>>>>> Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
>>>>>>>> INFO: Ping failed. Terminating the channel channel.
>>>>>>>> java.util.concurrent.TimeoutException: Ping started at
>>>>>>>> 1607120910589 hasn't completed by 1607121150590
>>>>>>>> at hudson.remoting.PingThread.ping(PingThread.java:134)
>>>>>>>> at hudson.remoting.PingThread.run(PingThread.java:90)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jigar R
>>>>>>>>
>>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Jenkins Users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to jenkinsci-use...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> --
>>>
>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Jenkins Users" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to
>>> jenkinsci-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> --
>> Un Saludo
>> Iván Fernández Calvo
>> https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> jenkinsci-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com
> <https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Un Saludo
Iván Fernández Calvo
https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAKo5Qrr7zpoT__4EeG-oXzPeW-518g56oPphTV0989yLgyr9wQ%40mail.gmail.com.
