Hey there - I think it might be worth posting the pipeline of your orchestration job that runs the 12000 builds for anyone to comment in more detail.
>From what I understand, if your pipeline is not orchestrating each build in a node section it will use something called a 'flyweight executor' on the Jenkins master. If this is happening, even though you have 100 agents your master may be heavily loaded running the pipeline - I guess each build allocates some Java objects that have to be GCed. If I can't find any documentation, I tend to go onto Github to look at the source code to understand what the Jenkins pipeline is doing behind the scenes: https://github.com/jenkinsci/workflow-cps-plugin. I ran a test on a system where I had an orchestrator kicking off 1000 builds every 4 minutes on a schedule. After a few days the Jenkins service stopped responding (Java memory issues). One thing I have found I needed to pay attention to was file and process limits as documented here: https://support.cloudbees.com/hc/en-us/articles/204231510-Memory-problem-unable-to-create-new-native-thread- All the best, --Bill On Thursday, 23 February 2017 21:33:42 UTC, Chris Overend wrote: > > So not sure if this is a Jenkins limitation or pipeline. > The jobs never exceeded available resources. > The garbage collection was stable. > > So why did it lock-up? > > It did say I used > > - 2950 million active threads > - 350 threads > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a504182f-ebf8-46a6-b343-03f639fefc8a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
