Dear Rob, Many tks for your long answer. As far as I understand the WAN distributed infrastructure is due to several factors : JUNIT tests and some war deploy on servers are done next on the same LAN as the slaves most of the time. But then I'm wondering if we can't have several master, one per site as you suggest... problem is that some projects are spread over several sites, and all consolidates on a central Nexus. I have to check with R&D team if such architecture make sens to them.
I've now to consider migration scenario for upgrade : migrate everything on the same server or create from scratch a new instance with new set of plugin and migrate only validated jobs.... with so much slaves and jobs, I have to organize a steering committee with users and find the best approach... ./Fred Le mardi 19 août 2014 16:21:06 UTC+2, Rob Mandeville a écrit : > > I used to run a system with about as many executors, but all in one > server room. > > > > First off, is there a good reason to run the slaves on a WAN? I > understand that your users are spread throughout Europe, but why should > your builds be? Users generally only need to talk to the Jenkins server, > not every piece of the build farm. Would it be possible to simply ship all > of your racks to one building? > > > > If you need a geographically distributed build farm, you should see how > much you can localize resources to a single farm. Of your three SCM > systems, Git is distributed, so it’s preferable. I suspect you don’t get > to choose, however. When using ClearCase, especially with a distributed > build farm, make sure to use snapshot views. You do not want every single > file read to be a transcontinental event. > > > > Jenkins requires a permanent connection to the slave nodes. If your WAN > is even a little bit flaky, you could lose slaves easily. Consider putting > a separate Jenkins server at each site, and then having a master one for > user interface (where the configuration for “build project Foo” is “Tell > Jenkins in Frankfurt to build project Foo”). This will be easier if you > can isolate various projects to a single site each. > > > > Per the memory leak on the server: do as little as possible on the > server’s address space. See if you can reduce the number of master nodes > to zero, so everything is done on the slaves. If you aren’t already, > consider using Java 7 or Java 8, each of which has its own strategy to GC > PermGenSpace. > > > > Your Jenkins server(s) should be well-connected and have plenty of fast > disk. Logs and artifacts end up being sent to the server in real time. > Also, all of the data (except for the logs and artifacts) from the build > runs it is aware of goes into memory on the server. To reduce this, set > jobs to erase old builds—do something like keep the last 100 runs, or the > last 30 days of runs. > > > > If you’ve got a farm this big, you may want to go pro. While Cloudbees’ > bread and butter is serving you build farms on the cloud, they also sell > CloudBees Enterprise. This is a support contract plus some extra plugins > not available for general release. Said plugins are designed for > controlling large deployments like yours. My previous employer used it, > and liked it. It is a bit expensive, and the license fee is per server and > per executor. What we didn’t try is a new product called Jenkins Ops > Center. This appears to be able to tie multiple Jenkins servers together. > They are worth investigating. > > > > Finally, if you’re considering upgrades, you don’t want the Latest and > Greatest. Get the LTS (Long Term Support) release; currently, that’s > 1.565.1. > > > > Hope this helps! > > > > --Rob > > > > *From:* jenkins...@googlegroups.com <javascript:> [mailto: > jenkins...@googlegroups.com <javascript:>] *On Behalf Of *Frederic Meyrou > *Sent:* Tuesday, August 19, 2014 8:09 AM > *To:* jenkins...@googlegroups.com <javascript:> > *Subject:* Large scale Jenkins installation : Help! > > > > Dear mates, > > > > I'm recently in charge of quite big Jenkins configuration based on a > central master (no build on master) with : > > > > >7000 Jobs > > 136 active Slaves mostly tied build > > 150-200 Total Slaves > > ~450 potential executors > > > > Users are on many R&D sites in Europe, slaves are spread over the WAN each > site has a Nexus proxy next to the slaves. > > SCM servers (SVN/ClearCase/Git) are not on the same LAN... > > > > At the moment the console is quite slow and we had recently memory trouble > that end-up adding 4 more Gb of RAM to the server and review the JVM setup > > below. At the moment it looks better, but I can't be sure because most > builds are schedule at night... I installed the monitoring plugin. > > The strange thing is that Jenkins is using more memory on the Windows > server then just the total of XMX + MaxPermSize (7+1 Gb), and ends-up using > more > > then 10 Gb... any idea why? > > I found a memory leak affecting the PermGen memory, the number of loaded > classes rise continuously and fills-up PermGem memory at the same time. I > have to restart Jenkins weekly. > > It saw also slave build making surefire reports over 200Mb going back to > the server. > > I'm also convinced that I should migrate to a more recent version of > Jenkins, but with such a big config and so much users, it's not so easy! > > > > *What do you think of such a configuration?* Do you think the server is > well sized or under-sized, should I use several master? Any TIPs for large > scale > > administration is welcome! > > > > *My configuration :* > > > VMWare VM > > Win 2K8R2 64bits > > 12 Gb VRAM > > 4 VCPU Xeon X5660 2,8 Ghz > > JENKINS : 1.424.1 > > TOMCAT : 6.0.29 > > JDK : 1.7.0_03 > > > > Tomcat JVM Setup as a NT Service : > > -Xms7g -Xmx7g -Xss128kb > > -XX:+UseLargePages > > -XX:PermSize=1g > > -XX:MaxPermSize=1g > > -XX:+UseConcMarkSweepGC > > -XX:+UseParNewGC > > -XX:+CMSIncrementalMode > > -XX:+CMSIncrementalPacing > > -XX:CMSInitiatingOccupancyFraction=33 > > -XX:CMSIncrementalDutyCycleMin=10 > > -XX:CMSIncrementalDutyCycle=50 > > -XX:ParallelGCThreads=8 > > -XX:MaxGCPauseMillis=2000 > > -XX:GCTimeRatio=10 > > -XX:+DisableExplicitGC > > -Xloggc:K:\apache-tomcat-6.0.29\logs\gc.log > > -verbose:gc > > -XX:+PrintGCDetails > > -XX:+PrintGCDateStamps > > -XX:+UseCompressedOops > > -Duser.language=US > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jenkinsci-use...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > Click here > <https://www.mailcontrol.com/sr/J8dQMICqEuPGX2PQPOmvUikZGg4PuPFT4xWhPg7vZpYK1zIIcgjxuCxk1v1nZPoJBQ2GHV9mBJsJwVIOzJj0SQ==> > > to report this email as spam. > > ------------------------------ > This e-mail and the information, including any attachments it contains, > are intended to be a confidential communication only to the person or > entity to whom it is addressed and may contain information that is > privileged. If the reader of this message is not the intended recipient, > you are hereby notified that any dissemination, distribution or copying of > this communication is strictly prohibited. If you have received this > communication in error, please immediately notify the sender and destroy > the original message. > > Thank you. > > Please consider the environment before printing this email. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
