On 23/02/12 07:26, Steve Shipway wrote:
Our Puppet system here is currently managing about 500 nodes. We
anticipate about 1000 eventually.
I have had to reduce the client frequency to once every 4 hours; it
seems that the maximum that can be handled by a single (dual-CPU, 8GB)
puppet master is 200 nodes. After that, performance drops quickly and
I notice many failed manifests. This is with Puppet 2.7.10 on the master.
Hi Steve,
Excuse the slight change in topic but I'm interested in the performance
stats you posted. I run Puppet 2.7.5 on a 4 CPU 4 GiB RAM KVM virtual
machine. I use Puppet Commander to evenly distribute runs and my
interval time works out to be around 15 minutes for 230 odd hosts, as
per the timestamps between MCollective discoveries below:
[root@gs2puppet01 ~]# grep Found /var/log/puppetcommander.log | tail
I, [2012-02-23T06:46:12.218853 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T06:57:59.009689 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T07:09:49.237810 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T07:21:39.435558 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T07:33:26.554525 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T07:45:59.550541 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T07:57:51.013245 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T08:12:10.915308 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T08:24:16.383794 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I, [2012-02-23T08:37:03.750438 #28284] INFO -- : Found 231 puppet
nodes, sleeping for ~3 seconds between runs
I allow 10 Agents to run concurrently however my catalogs are very very
light, less than a second to compile:
[root@gs2puppet01 ~]# grep 'Compiled catalog' /var/log/messages | awk
'{sum+=$14} END {print sum/NR}'
0.750115
How big are your Puppet manifests so that you've had to drop the run
time down to 4 hours? Have you considered the use of MCollective and
Puppet Commander to spread your load out more?
-Luke
We've bought a copy of ProPuppet (as Jeff Watts recommended) and
we're planning to make a distributed system as instructed in there --
one puppet dashboard/report server, multiple puppet master servers,
and one dev server. Puppet configurations held is subversion and
synchronised on all puppet masters, which would themselves be behind a
load balancer. This is still in the planning stage, though.
I'd be interested in hearing your experiences in managing your
extra-large system; I can also share our experiences in how we
implemented and manage control of this system, if you'd like to
contact me off-list. When we first implemented, we engaged a Puppet
Labs consultant for a few days to help with the initial work. I can
definitely recommend doing this if you've no puppet experience, as one
place Puppet lacks is documentation!
Steve
*Steve Shipway*
University of Auckland ITS
/UNIX Systems Design Lead/
s.ship...@auckland.ac.nz <mailto:s.ship...@auckland.ac.nz>
Ph: +64 9 373 7599 ext 86487
//
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
Luke Bigum
Information Systems
luke.bi...@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
The information in this e-mail and any attachment is confidential and is
intended only for the named recipient(s). The e-mail may not be disclosed or
used by any person other than the addressee, nor may it be copied in any way.
If you are not a named recipient please notify the sender immediately and
delete any copies of this message. Any unauthorized copying, disclosure or
distribution of the material in this e-mail is strictly forbidden. Any view or
opinions presented are solely those of the author and do not necessarily
represent those of the company.
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
