I used to run a system with about as many executors, but all in one server room.
First off, is there a good reason to run the slaves on a WAN? I understand that your users are spread throughout Europe, but why should your builds be? Users generally only need to talk to the Jenkins server, not every piece of the build farm. Would it be possible to simply ship all of your racks to one building? If you need a geographically distributed build farm, you should see how much you can localize resources to a single farm. Of your three SCM systems, Git is distributed, so it’s preferable. I suspect you don’t get to choose, however. When using ClearCase, especially with a distributed build farm, make sure to use snapshot views. You do not want every single file read to be a transcontinental event. Jenkins requires a permanent connection to the slave nodes. If your WAN is even a little bit flaky, you could lose slaves easily. Consider putting a separate Jenkins server at each site, and then having a master one for user interface (where the configuration for “build project Foo” is “Tell Jenkins in Frankfurt to build project Foo”). This will be easier if you can isolate various projects to a single site each. Per the memory leak on the server: do as little as possible on the server’s address space. See if you can reduce the number of master nodes to zero, so everything is done on the slaves. If you aren’t already, consider using Java 7 or Java 8, each of which has its own strategy to GC PermGenSpace. Your Jenkins server(s) should be well-connected and have plenty of fast disk. Logs and artifacts end up being sent to the server in real time. Also, all of the data (except for the logs and artifacts) from the build runs it is aware of goes into memory on the server. To reduce this, set jobs to erase old builds—do something like keep the last 100 runs, or the last 30 days of runs. If you’ve got a farm this big, you may want to go pro. While Cloudbees’ bread and butter is serving you build farms on the cloud, they also sell CloudBees Enterprise. This is a support contract plus some extra plugins not available for general release. Said plugins are designed for controlling large deployments like yours. My previous employer used it, and liked it. It is a bit expensive, and the license fee is per server and per executor. What we didn’t try is a new product called Jenkins Ops Center. This appears to be able to tie multiple Jenkins servers together. They are worth investigating. Finally, if you’re considering upgrades, you don’t want the Latest and Greatest. Get the LTS (Long Term Support) release; currently, that’s 1.565.1. Hope this helps! --Rob From: jenkinsci-users@googlegroups.com [mailto:jenkinsci-users@googlegroups.com] On Behalf Of Frederic Meyrou Sent: Tuesday, August 19, 2014 8:09 AM To: jenkinsci-users@googlegroups.com Subject: Large scale Jenkins installation : Help! Dear mates, I'm recently in charge of quite big Jenkins configuration based on a central master (no build on master) with : >7000 Jobs 136 active Slaves mostly tied build 150-200 Total Slaves ~450 potential executors Users are on many R&D sites in Europe, slaves are spread over the WAN each site has a Nexus proxy next to the slaves. SCM servers (SVN/ClearCase/Git) are not on the same LAN... At the moment the console is quite slow and we had recently memory trouble that end-up adding 4 more Gb of RAM to the server and review the JVM setup below. At the moment it looks better, but I can't be sure because most builds are schedule at night... I installed the monitoring plugin. The strange thing is that Jenkins is using more memory on the Windows server then just the total of XMX + MaxPermSize (7+1 Gb), and ends-up using more then 10 Gb... any idea why? I found a memory leak affecting the PermGen memory, the number of loaded classes rise continuously and fills-up PermGem memory at the same time. I have to restart Jenkins weekly. It saw also slave build making surefire reports over 200Mb going back to the server. I'm also convinced that I should migrate to a more recent version of Jenkins, but with such a big config and so much users, it's not so easy! What do you think of such a configuration? Do you think the server is well sized or under-sized, should I use several master? Any TIPs for large scale administration is welcome! My configuration : VMWare VM Win 2K8R2 64bits 12 Gb VRAM 4 VCPU Xeon X5660 2,8 Ghz JENKINS : 1.424.1 TOMCAT : 6.0.29 JDK : 1.7.0_03 Tomcat JVM Setup as a NT Service : -Xms7g -Xmx7g -Xss128kb -XX:+UseLargePages -XX:PermSize=1g -XX:MaxPermSize=1g -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSInitiatingOccupancyFraction=33 -XX:CMSIncrementalDutyCycleMin=10 -XX:CMSIncrementalDutyCycle=50 -XX:ParallelGCThreads=8 -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=10 -XX:+DisableExplicitGC -Xloggc:K:\apache-tomcat-6.0.29\logs\gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseCompressedOops -Duser.language=US -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com<mailto:jenkinsci-users+unsubscr...@googlegroups.com>. For more options, visit https://groups.google.com/d/optout. Click here<https://www.mailcontrol.com/sr/J8dQMICqEuPGX2PQPOmvUikZGg4PuPFT4xWhPg7vZpYK1zIIcgjxuCxk1v1nZPoJBQ2GHV9mBJsJwVIOzJj0SQ==> to report this email as spam. ________________________________ This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
