I see some serialization fails and this breaks the channel, the plugin that
causes the exception seems https://github.com/jenkinsci/tasks-plugin, and
the `[Deprecated] Scan workspace for open tasks` I think matters, this
plugins has been integrated into
https://github.com/jenkinsci/warnings-ng-plugin and
https://github.com/jenkinsci/analysis-model
ERROR: Step ‘[Deprecated] Scan workspace for open tasks’ aborted due
to exception:
java.io.StreamCorruptedException: invalid type code: 6D
...
at hudson.plugins.tasks.TasksPublisher.perform(TasksPublisher.java:182)
at
hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:69)
at
hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:298)
at jenkins.tasks.SimpleBuildStep.perform(SimpleBuildStep.java:112)
at
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1919)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:428)
El jue, 4 feb 2021 a las 14:58, Jigar R (<[email protected]>)
escribió:
> Hello Ivan,
>
> Attached the logs in the email
>
> - jenkins.log - jenkins build output
> - Jenkins-agent.log - output from jenkins ssh agent
>
> Jenkins SSH agent was created with following:
>
> -
> - launch method : launch agents via ssh
> - JavaPath: $JAVA_HOME
> - JVM options: -Xmx2048m -Xms2048m
> - Use TCP_NODELAY flag on the ssh connection - enabled
>
> Environment information
>
>
> - Jenkins v2.249.1
> - RH6
> - SSH agent plugin 1.20
> - SSH slaves plugin 1.30.4
>
>
> On Tuesday, February 2, 2021 at 5:27:07 PM UTC-5 [email protected]
> wrote:
>
>> Weird, Could you share a screencapture of what you configurate? Also the
>> whole exception those lines alone mean nothing. Know the version of Jenkins
>> and the version of the SSH build agents plugins you use can help, the
>> memory you have in your agents, if they are bare metal or cloud. In overall
>> if you want help please provide more context.
>>
>> El mar, 2 feb 2021 a las 22:52, Jigar R (<[email protected]>)
>> escribió:
>>
>>>
>>>
>>> On Tuesday, February 2, 2021 at 12:20:19 PM UTC-5 [email protected]
>>> wrote:
>>>
>>>> >I have 2 different kind of jenkins agents.
>>>> >1. java web start
>>>> >2. ssh
>>>> >If the memory was issue then wouldn't it fail on both cases?
>>>>
>>>> Not necessarily, starting by the point that are different ways to
>>>> establish the connection, the JNLP agents could not update the remoting jar
>>>> file (depends on your configurations), so you can be running different
>>>> versions of remoting. I agree with Jeff looks like an OOM issue, review my
>>>> comments at
>>>> https://groups.google.com/g/jenkinsci-users/c/nD3s06hSUXE/m/BQKk5GSYBwAJ
>>>> my recommendation is to fix the mem for the remoting process to 1024M
>>>> (-Xmx1024m -Xms1024m) see if the issue disappear or change, if disappear,
>>>> you would have to adjust the remoting process memory to the right one
>>>> between 256M-1024M, to use 512M usually is safe a not too much (but depends
>>>> on your agents' memory we do not know how much they have)
>>>>
>>>
>>> I created SSH agent with -Xmx1024m -Xms1024m.
>>> I do see bunch of Warnings about "Ping failed. Terminating the channel
>>> channel".
>>> I got following error:
>>> "
>>> WARNING: Failed to send back a reply to te rquest
>>> hudson.remoting.Request$....
>>> java.io.IOException: Resource temporarily unavailable
>>> at java.io.FileOutputStream.writeBytes(Native method)
>>> .....
>>> "
>>>
>>>
>>>>
>>>> El lunes, 1 de febrero de 2021 a las 23:52:19 UTC+1,
>>>> [email protected] escribió:
>>>>
>>>>> On Monday, January 4, 2021 at 12:37:52 PM UTC-5 [email protected]
>>>>> wrote:
>>>>>
>>>>>> When I saw something like this in the past, it was because the
>>>>>> process was running out of resources. Specifically when running
>>>>>> SpotBugs, I
>>>>>> got out-of-memory errors. I had to modify the pom to allocate more
>>>>>> memory.
>>>>>>
>>>>> I have 2 different kind of jenkins agents.
>>>>> 1. java web start
>>>>> 2. ssh
>>>>> If the memory was issue then wouldn't it fail on both cases?
>>>>>
>>>>> I do see "Agent went offline during build Connection was broker:
>>>>> java.io.StreamCorruptedException: invalid stream header:".
>>>>>
>>>>>> I recommend ensuring your build runs normally on the agent without
>>>>>> the additional complexities of the Jenkins environment. This might show
>>>>>> where additional resources are needed. If that all passes, then continue
>>>>>> on
>>>>>> to the additional troubleshooting steps involving the Jenkins controller
>>>>>> and agent.
>>>>>>
>>>>>> Jeff Thompson
>>>>>> On 12/30/20 3:40 PM, [email protected] wrote:
>>>>>>
>>>>>> Interestingly, whenever Jenkins agent is executing PMD/Findbugs task,
>>>>>> it runs into " Resource temporarily unavailable".
>>>>>> I have
>>>>>>
>>>>>> - verified TCP parameters
>>>>>> i.e. tcp_keepalive_time, tcp_keepalive_intvl, tcp_keepalive_probes
>>>>>> & tcp_fin_timeout ; All of them have higher values then what cloudbees
>>>>>> recommended at
>>>>>>
>>>>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected
>>>>>> - set heap space to 2-4Gb
>>>>>> - disabled ping thread from Jenkins agent by setting
>>>>>> "-Dhudson.remoting.Launcher.pingIntervalSec=-1"
>>>>>>
>>>>>> Please let me know if there is anything else I should consider.
>>>>>> On Friday, December 11, 2020 at 12:11:39 PM UTC-5 [email protected]
>>>>>> wrote:
>>>>>>
>>>>>>> The most common issue is related to disconnections because there is
>>>>>>> no traffic between the Jenkins instance and the agent, for that, you
>>>>>>> have
>>>>>>> to tune the TCP stack of your OS (see
>>>>>>> https://support.cloudbees.com/hc/en-us/articles/115001369667-dedicated-SSH-agents-formerly-slaves-get-disconnected),
>>>>>>> or enable the keepalive option in the SSH protocol this can be
>>>>>>> configured
>>>>>>> by setting ClientAliveInterval or TCPKeepAlive on the SSH server
>>>>>>> (/etc/ssh/sshd_config), also by setting ServerAliveInterval or
>>>>>>> TCPKeepAlive
>>>>>>> options for the user connection (/etc/ssh/ssh_config or
>>>>>>> ~/.ssh/ssh_config)
>>>>>>>
>>>>>>> https://www.freebsd.org/cgi/man.cgi?sshd_config(5)
>>>>>>> https://www.freebsd.org/cgi/man.cgi?ssh_config(5)
>>>>>>>
>>>>>>> Also, check you follow the best practices to configure your SSH
>>>>>>> agents and enable SSH verbose log output in your service (see
>>>>>>> https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md
>>>>>>> )
>>>>>>>
>>>>>>> El miércoles, 9 de diciembre de 2020 a las 17:53:19 UTC+1,
>>>>>>> [email protected] escribió:
>>>>>>>
>>>>>>>> Hello Jenkins community,
>>>>>>>>
>>>>>>>> I have setup Jenkins agents over SSH and this agent frequently goes
>>>>>>>> offline with Resource temporarily unavailable error. I had to
>>>>>>>> configure my
>>>>>>>> Jenkins agents to communicate over JNLP. Any advise on how to fix this
>>>>>>>> issue?
>>>>>>>>
>>>>>>>> Jenkins v2.249.1
>>>>>>>>
>>>>>>>> [12/04/20 15:23:00] [SSH] Checking java version of java
>>>>>>>> [12/04/20 15:23:01] [SSH] java -version returned 1.8.0_202.
>>>>>>>> [12/04/20 15:23:01] [SSH] Starting sftp client.
>>>>>>>> [12/04/20 15:23:03] [SSH] Remote file system root $JENKINS_SSH_DATA
>>>>>>>> does not exist. Will try to create it...
>>>>>>>> [12/04/20 15:23:03] [SSH] Copying latest remoting.jar...
>>>>>>>> [12/04/20 15:23:03] [SSH] Copied 1,521,553 bytes.
>>>>>>>> Expanded the channel window size to 4MB
>>>>>>>> [12/04/20 15:23:03] [SSH] Starting agent process: cd
>>>>>>>> "$JENKINS_SSH_DATA" && java -jar remoting.jar -workDir
>>>>>>>> $JENKINS_SSH_DATA
>>>>>>>> -jar-cache $JENKINS_SSH_DATA/remoting/jarCache
>>>>>>>> Dec 04, 2020 3:23:29 PM
>>>>>>>> org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
>>>>>>>> INFO: Using $JENKINS_SSH_DATA/remoting as a remoting work directory
>>>>>>>> Dec 04, 2020 3:23:29 PM
>>>>>>>> org.jenkinsci.remoting.engine.WorkDirManager setupLogging
>>>>>>>> INFO: Both error and output logs will be printed to
>>>>>>>> $JENKINS_SSH_DATA/remoting
>>>>>>>> <===[JENKINS REMOTING CAPACITY]===>channel started
>>>>>>>> Remoting version: 4.5
>>>>>>>> This is a Unix agent
>>>>>>>> Evacuated stdout
>>>>>>>> Agent successfully connected and online
>>>>>>>> The Agent is connected, disconnect it before to try to connect it
>>>>>>>> again.
>>>>>>>> Dec 04, 2020 3:33:58 PM org.eclipse.jgit.util.FS
>>>>>>>> discoverGitSystemConfig
>>>>>>>> WARNING: Exception caught during execution of command '[git,
>>>>>>>> config, --system, --edit]' in '$GIT_PATH/bin', return code '128', error
>>>>>>>> message 'fatal: Invalid path '$GIT_PATH/etc': No such file or directory
>>>>>>>> '
>>>>>>>> Dec 04, 2020 3:33:58 PM
>>>>>>>> org.eclipse.jgit.util.FS$FileStoreAttributes saveToConfig
>>>>>>>> WARNING: locking FileBasedConfig[$JENKINS_PATH/.config/jgit/config]
>>>>>>>> failed after 5 retries
>>>>>>>> Dec 04, 2020 3:33:59 PM
>>>>>>>> org.jenkinsci.remoting.util.AnonymousClassWarnings warn
>>>>>>>> WARNING: Attempt to (de-)serialize anonymous class
>>>>>>>> com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritTriggerBuildChooser$1;
>>>>>>>> see:
>>>>>>>> https://jenkins.io/redirect/serialization-of-anonymous-classes/
>>>>>>>> Dec 04, 2020 3:34:24 PM hudson.remoting.Request$2 run
>>>>>>>> WARNING: Failed to send back a reply to the request
>>>>>>>> hudson.remoting.Request$2@493c5a4e
>>>>>>>> java.io.IOException: Resource temporarily unavailable
>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:313)
>>>>>>>> at
>>>>>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)
>>>>>>>> at
>>>>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
>>>>>>>> at hudson.remoting.Channel.send(Channel.java:766)
>>>>>>>> at hudson.remoting.Request$2.run(Request.java:388)
>>>>>>>> at
>>>>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>
>>>>>>>> Dec 04, 2020 5:25:30 PM hudson.remoting.Request$2 run
>>>>>>>> WARNING: Failed to send back a reply to the request
>>>>>>>> hudson.remoting.Request$2@2511e2d4
>>>>>>>> java.io.IOException: Resource temporarily unavailable
>>>>>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>>>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>>>>>>> at
>>>>>>>> hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:88)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:90)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:85)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:54)
>>>>>>>> at java.io.OutputStream.write(OutputStream.java:75)
>>>>>>>> at
>>>>>>>> hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
>>>>>>>> at
>>>>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)
>>>>>>>> at hudson.remoting.Channel.send(Channel.java:766)
>>>>>>>> at hudson.remoting.Request$2.run(Request.java:388)
>>>>>>>> at
>>>>>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>
>>>>>>>> ERROR: Connection terminated
>>>>>>>> java.io.StreamCorruptedException: invalid stream header: 00025B42
>>>>>>>> at
>>>>>>>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:866)
>>>>>>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
>>>>>>>> at
>>>>>>>> hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
>>>>>>>> at hudson.remoting.Command.readFrom(Command.java:142)
>>>>>>>> at hudson.remoting.Command.readFrom(Command.java:128)
>>>>>>>> at
>>>>>>>> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
>>>>>>>> at
>>>>>>>> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
>>>>>>>> Dec 04, 2020 5:32:30 PM hudson.slaves.ChannelPinger$1 onDead
>>>>>>>> INFO: Ping failed. Terminating the channel channel.
>>>>>>>> java.util.concurrent.TimeoutException: Ping started at
>>>>>>>> 1607120910589 hasn't completed by 1607121150590
>>>>>>>> at hudson.remoting.PingThread.ping(PingThread.java:134)
>>>>>>>> at hudson.remoting.PingThread.run(PingThread.java:90)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jigar R
>>>>>>>>
>>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Jenkins Users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/jenkinsci-users/030aced0-ba6e-4012-a60a-58208d350544n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> --
>>>
>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Jenkins Users" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/jenkinsci-users/615a4333-587c-4ea2-a11f-fea45fe38a56n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> --
>> Un Saludo
>> Iván Fernández Calvo
>> https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/nD3s06hSUXE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com
> <https://groups.google.com/d/msgid/jenkinsci-users/32d22541-1143-4c96-be3b-23c10bb5619cn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
Un Saludo
Iván Fernández Calvo
https://www.linkedin.com/in/iv%C3%A1n-fern%C3%A1ndez-calvo-21425033
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/CAKo5Qrr7zpoT__4EeG-oXzPeW-518g56oPphTV0989yLgyr9wQ%40mail.gmail.com.
