Could it be issue with the virtual memory in the jenkins server? Because Jenkins does consume a lot of virtual memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13565 jenkins 20 0 12.641g 0.011t 13552 S 0.000 56.62 625:25.18 /usr/bin/java -XX:+UseG1GC -Xmx10g It has amassed 12G of virtual memory. It is a problem with java and glibc that can be remedied with MALLOC_ARENA_MAX=1 I have tried adding ENVIRONMENT to /etc/systemd/system/jenkins.service, but it is not set before running jenkins. https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used/28935176#28935176 tirsdag 9. juli 2019 13.24.18 UTC+2 skrev Sverre Moe følgende: > > > Since we don't use 32bit, the reason must be > - the virtual memory of the OS has been fully depleted > > How can I check for this, and remedy it? > > fredag 5. juli 2019 03.17.38 UTC+2 skrev Jan Monterrubio følgende: > >> Correct me if I’m wrong but I don’t think increasing heap size will >> actually affect your ability to create more native threads. >> >> See this for a possible explanation: >> https://plumbr.io/outofmemoryerror/unable-to-create-new-native-thread >> >> On Thu, Jul 4, 2019 at 16:03 Baptiste Mathus <m...@batmat.net> wrote: >> >>> Did you enable GC logging to have a better understanding of the profile >>> of your memory consumption? If not, I would recommend you do it first and >>> analyze them. >>> https://jenkins.io/blog/2016/11/21/gc-tuning/ explained this part (and >>> much more) quite well. >>> >>> Then, once you understand better when it crashes, possibly you'll want >>> to analyze a heap dump to see what is causing the problem. >>> >>> Cheers >>> >>> >>> Le mar. 2 juil. 2019 à 15:30, Sverre Moe <sver...@gmail.com> a écrit : >>> >>>> Today it has been chaotic. >>>> Several build agents disconnected >>>> >>>> Unexpected termination of the channel >>>> >>>> Many builds failed because of Memory error. >>>> >>>> I have tried restarting Jenkins several times today. >>>> >>>> Anyone have any suggestions? >>>> >>>> tirsdag 2. juli 2019 14.34.25 UTC+2 skrev Sverre Moe følgende: >>>>> >>>>> We have assigned 8GB of memory to our Jenkins instance. >>>>> JAVA_OPTIONS=-Xmx8g >>>>> >>>>> Still we experience memory issues after a while running. >>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>> >>>>> We have: >>>>> aprox 40 connected build agents >>>>> aprox 400 pipeline jobs >>>>> >>>>> We have a test Jenkins instance running with the same jobs, this one >>>>> connects to the same build agents (though on a different home directory). >>>>> >>>>> Lately we have been getting disconnected build agents, that we cannot >>>>> get up again without restarting Jenkins. >>>>> >>>>> Can we assign more memory to a build agent? Would it have any affect >>>>> on this issue? >>>>> >>>>> This we got from one of our latest Pipeline builds that failed on a >>>>> sh("find **** -exec ***") step. It failed on that build agent that is now >>>>> disconnected. >>>>> >>>>> >>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>> at java.lang.Thread.start0(Native Method) >>>>> at java.lang.Thread.start(Thread.java:714) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368) >>>>> at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288) >>>>> at java.lang.UNIXProcess.lambda$new$2(UNIXProcess.java:258) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:257) >>>>> at java.lang.ProcessImpl.start(ProcessImpl.java:134) >>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) >>>>> at hudson.Proc$LocalProc.<init>(Proc.java:249) >>>>> Also: java.io.IOException: error=11, Resource temporarily unavailable >>>>> >>>>> >>>>> >>>>> >>>>> SEVERE: Unexpected error when retrieving changeset >>>>> hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M >>>>> "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> % >>>>> ai%ncommitter %cN <%cE> %ci%n%n%w(76,4,4)%s%n%n%b" -n 1 >>>>> b2c871830a03ea5f2fd2b21245afb09d51d69686 in >>>>> /home/build/jenkins/workspace/ >>>>> project_user_work >>>>> at >>>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1012) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) >>>>> >>>>> >>>>> at hudson.remoting.UserRequest.perform(UserRequest.java:212) >>>>> at hudson.remoting.UserRequest.perform(UserRequest.java:54) >>>>> at hudson.remoting.Request$2.run(Request.java:369) >>>>> at >>>>> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) >>>>> >>>>> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>>>> >>>>> >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>>>> >>>>> >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote >>>>> call to master-sles12.3-x86_64_3 >>>>> at >>>>> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) >>>>> at >>>>> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) >>>>> >>>>> >>>>> at hudson.remoting.Channel.call(Channel.java:955) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) >>>>> >>>>> >>>>> at >>>>> sun.reflect.GeneratedMethodAccessor678.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> >>>>> >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at >>>>> org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) >>>>> >>>>> >>>>> at com.sun.proxy.$Proxy104.execute(Unknown Source) >>>>> at >>>>> io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159) >>>>> >>>>> >>>>> at >>>>> io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140) >>>>> >>>>> at >>>>> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingS >>>>> tepExecution.java:47) >>>>> >>>>> Jul 01, 2019 11:51:12 AM >>>>> hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler >>>>> >>>>> uncaughtException >>>>> SEVERE: A thread (Timer-9692/111139) died unexpectedly due to an >>>>> uncaught exception, this may leave your Jenkins in a bad way and >>>>> is usually indicative of a bug in the code. >>>>> java.lang.OutOfMemoryError: unable to create new native thread >>>>> at java.lang.Thread.start0(Native Method) >>>>> at java.lang.Thread.start(Thread.java:714) >>>>> at java.util.Timer.<init>(Timer.java:160) >>>>> at java.util.Timer.<init>(Timer.java:132) >>>>> at >>>>> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437) >>>>> >>>>> >>>>> at >>>>> org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299) >>>>> >>>>> >>>>> at java.util.TimerThread.mainLoop(Timer.java:555) >>>>> at java.util.TimerThread.run(Timer.java:505) >>>>> >>>>> INFO: Ping failed. Terminating the channel master-sles12.3-x86_64_3. >>>>> java.util.concurrent.TimeoutException: Ping started at 1561982408948 >>>>> hasn't completed by 1561982648948 >>>>> at hudson.remoting.PingThread.ping(PingThread.java:134) >>>>> at hudson.remoting.PingThread.run(PingThread.java:90) >>>>> >>>>> Jul 01, 2019 2:04:11 PM >>>>> hudson.remoting.SynchronousCommandTransport$ReaderThread run >>>>> INFO: I/O error in channel master-sles12.3-x86_64_3 >>>>> java.io.IOException: Unexpected termination of the channel >>>>> WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Temp >>>>> Space >>>>> >>>>> Jul 01, 2019 2:04:11 PM >>>>> hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed >>>>> WARNING: Failed to monitor master-sles12.3-x86_64_3 for Free Swap >>>>> Space >>>>> >>>>> >>>>> >>>>> The latest problem we got. It did not take down the build node. On all >>>>> occasions of this problem it happens when the Pipeline is doing some IO >>>>> on >>>>> the Jenkins master. Here we manually restart the build again, and it >>>>> builds >>>>> fine. >>>>> >>>>> Running on Jenkins <https://build-ci.spacetec.no:8443/computer/(master)/> >>>>> in /var/lib/jenkins/workspace/project_master[Pipeline] { >>>>> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline] >>>>> parallel >>>>> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline] >>>>> { (Branch: Setup) >>>>> <https://build-ci.spacetec.no:8443/job/mcap_hrdfep/job/JPSS/75/console#>[Pipeline] >>>>> End of Pipelinejava.lang.OutOfMemoryError: unable to create new native >>>>> thread >>>>> at java.lang.Thread.start0(Native Method) >>>>> at java.lang.Thread.start(Thread.java:714) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1366) >>>>> at >>>>> com.google.common.eventbus.AsyncEventBus.dispatch(AsyncEventBus.java:90) >>>>> at >>>>> com.google.common.eventbus.AsyncEventBus.dispatchQueuedEvents(AsyncEventBus.java:81) >>>>> at com.google.common.eventbus.EventBus.post(EventBus.java:264) >>>>> at >>>>> org.jenkinsci.plugins.pubsub.GuavaPubsubBus$1.publish(GuavaPubsubBus.java:70) >>>>> at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:141) >>>>> at >>>>> io.jenkins.blueocean.events.PipelineEventListener.publishEvent(PipelineEventListener.java:196) >>>>> at >>>>> io.jenkins.blueocean.events.PipelineEventListener.onNewHead(PipelineEventListener.java:85) >>>>> at >>>>> org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) >>>>> at >>>>> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) >>>>> at >>>>> org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) >>>>> at >>>>> hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) >>>>> at >>>>> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) >>>>> at >>>>> jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Finished: FAILURE >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to jenkins...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-users/6b1f3729-e456-41a9-a464-c63d061e2912%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jenkins...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5qokC%3D6vVcku1A6OXm%2Boi%3D03YzuSNQrCarDtCyCPy4pQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e49d203f-db67-41ae-b223-313c5535e537%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
