I only have the option to downgrade to 1.18 of Server Sent Events (SSE) Gateway Plugin I would have to download the 1.17 and manually downgrade it.
>From the discussions it seems I also need to downgrade BlueOcean to 1.17 > Downgrading to BlueOcean 1.17 (which in turn uses sse-gateway 1.17) appears to have resolved our issue This would be much more work. I would need to install all the BlueOcean 1.17 manually as I can only downgrade to 1.18.0 I might be willing to try this, even with the risk of https://issues.jenkins-ci.org/browse/JENKINS-51057 I am running SSE 1.19, and have previously recorded jstack from Jenkins PID. I could not find any EventDispatcher.retryProcessor \t onsdag 14. august 2019 15.38.17 UTC+2 skrev Devin Nusbaum følgende: > > I have not read the whole thread in detail, but the “Unable to create new > native thread” OutOfMemoryErrors from your original thread where one of the > stack traces involves > org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing > looks > like it could be related to > https://issues.jenkins-ci.org/browse/JENKINS-58684, which is a thread > leak caused by the SSE Gateway Plugin. You could try reverting the SSE > Gateway Plugin to version 1.17 to see if that helps, although that might > reintroduce a different, somewhat rarer memory leak ( > https://issues.jenkins-ci.org/browse/JENKINS-51057). To test my > hypothesis, if you are running SSE Gateway Plugin version 1.19, you can > collect thread dumps over time and see if you seem to have a large number > of threads named “EventDispatcher.retryProcessor” (unfortunately in version > 1.18 and below the threads are automatically named “Timer #n”, which is > less useful), which would confirm that you are hitting JENKINS-58684 > <https://issues.jenkins-ci.org/browse/JENKINS-58684>. > > The advice to stop building on master is definitely a good idea as well. > > On Aug 14, 2019, at 07:11, Sverre Moe <sver...@gmail.com <javascript:>> > wrote: > > We got an 30 minute free CloudBees support. It was too short to dig deeper > to find the problem, but the person I was talking to (after examining our > logs) mentioned what he thought was the problem and gave a suggestion. > > We should not use Jenkins master at all for builds (allocated with the > node("master") step). We had 15 Executors for Jenkins master. > > We could also try to Increase limits of hard nofile and nproc for jenkins > user, but the main recomondation was to remove all Executors for Jenkins > master. > > /etc/security/limits.conf > jenkins soft core unlimited > jenkins hard core unlimited > jenkins soft fsize unlimited > jenkins hard fsize unlimited > jenkins soft nofile 4096 > jenkins hard nofile 10240 #Was 8192 > jenkins soft nproc 30654 > jenkins hard nproc 60654 #Was 30654 > > > To remove Jenkins master Executors will take some time. We use Jenkins > master when we publish our build artifacts RPMs to our NFS file storage. > Since our RPM NFS is only attached to the Jenkins master it is not > possible at the moment. Unless we can use any other agent, then do a SCP > onto our Jenkins master with the RPM artifacts. > > > We had a few other circumstances where we used Jenkins master. Like > checking out a file to determine which build agent to actually use. These I > have already changed to use any available build agent instead. > > tirsdag 6. august 2019 09.48.50 UTC+2 skrev Sverre Moe følgende: >> >> Sadly I was mistaken. We do not use NFS for JENKINS_HOME. >> >> We do however use NFS for the location where builds copy the RPM build >> artifacts. >> >> mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende: >>> >>> Hi, >>> >>> Severe has another email thread open, I think it is the same Jenkins >>> instance >>> https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. >>> >>> I dunno what happens on your instance but probably it isn’t better that you >>> open another email thread with the description of your issue >> >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkins...@googlegroups.com <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com > > <https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d026b611-fab1-420a-a870-de5e6df291cf%40googlegroups.com.
