On Fri, 2011-12-16 at 04:56 -0800, Antonio Xanxess wrote: > I have a problem with my Puppet server, the server is overloaded and > processes need to add half of my farm nodes yet! > The server is hosted on a blade with the following characteristics: > Processors: 8 cores > Memory Ram: 12 Gb > Operating System: Debian Squeeze > Version of parcel: > 2.7.1-1 ~ bpo60 puppet 1 > puppet-common 2.7.1-1 ~ bpo60 1 > 2.7.1-1 ~ bpo60 1 puppetmaster > puppetmaster 2.7.1-1 ~ bpo60-1 common > puppet vim 2.7.1-1 ~ bpo60-1 > > The number of nodes that I have now is 1927 and I'm in the middle of > the deployment process. The nodes do not use the demon puppet, but I > created a cron task that runs dispersed to run in one hour intervals. > With these data the server is currently using on average 4 cores, but > the RAM is used almost entirely (it has 1 GB free of 12)
If your setup is consuming so much ram, that means your concurrency is too high. You might also benefit of using Ruby Enterprise Edition, because passenger might be able to share more memory between those ruby processes. Puppetmasters are mostly CPU bound. Since your server has 8 cores, having more than 8, 12 or 16 puppetmaster processes at one will not give you more horse power (yeah CPU is a finite quantity). What is your current concurrency? If your problem is that your CPU is constantly at 100% because of the puppet runs (ie too many clients checking-in at the same time), then I'm afraid you'll either: * need another server * increase the time between your client check in (every 60min in lieu of 30 for instance) * use of Mcollective Puppet Controller [1] * my experimental catalog caching [2] You can do some math to see if your near the limit: if you have 1927 nodes each checking-in every 30 minutes, that means you (if spread evenly) 1927/30=64 nodes checking in every minute. That's more than 1 node coming in per second. Since you have a concurrency of 8 (because of your 8 core) at max, you can accomodate 8 nodes at the same time. Since 64/8 = 8, you can accomodate 8 client per minute, which means each client must be serviced in less than 60/8=7.5s or you'll run into trouble. And you can even do better than math, you can benchmark your master (better a test server with the same capacity) [3] > I would like to know some technical expert for the optimization of > services, anything you can think of will help me because I am a bit > blocked .... > I've been thinking about creating another puppetmaster and use a load > balancer, but in my case would have security problems because the > connections are behind the balancer and servers unencrypted would be > in different physical locations, having to go unencrypted information > by the network. You can certainly load balance SSL connections without having to decrypt the content (ie a pure layer 3-4 load-balancer). I'm sure haproxy would be able to do that. If you go this route, make sure you have a standalone CA and point your clients to this standalone CA (that's way simpler than trying to replicate your CA between your masters). [1]: http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcollective.php [2]: http://www.masterzen.fr/2010/03/21/more-puppet-offloading/ [3]: http://www.masterzen.fr/2010/10/18/benchmarking-puppetmaster-stacks/ -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
