Frederic, Based on the size of your farm, you may want to start thinking about either load balancing your puppet master (nginx, ha proxy, SRV records, etc.) or possibly decentralizing your puppet deployments (masterless). Remember that all manifest compilations take place on the puppet master and not the clients. This means all collected resources, for all 350+ hosts must funnel through your puppet master, which then has to update the storedconfigs in puppetdb. This can be a significant amount of load for the puppet master, considering the size of your deployment.
So, if performance is a concern/issue with you, look into other methods of spreading the work load or decentralizing to get the performance you desire. *Michael Pawlak* Web Systems Administrator | Colovore LLC E: m...@colovore.com C: 408.316.2154 <http://www.colovore.com> On Wed, May 6, 2015 at 8:00 AM, SCHAER Frederic <frederic.sch...@cea.fr> wrote: > Hi Puppet users, > > > > I know this has been discussed several times, but I did not find the > information/fix I’m looking for, so here I go… > > I have a farm of ~370 servers. > > I have a single puppet master (12 physical cores, lots of RAM) on which I > deployed puppet 3.7.5 this week (not using r10k, that’s on our huge todo > list). > > I am running on Scientific Linux 6.2 (RHEL like) + Passenger + Foreman as > ENC. > > More than puppet 3.7.5, I deployed the saz/ssh (2.5.0) module (required by > the openstack modules), which in turns by default enables puppet exported > resources collection for the ssh keys. > > > > Since I was using a local hack with the generate function to achieve the > same goal, I let the ssh keys collection in place… and it worked OK until I > deployed that in production . > > I then faced huge Passenger overloads, and nearly all puppet runs failed > until I raised the PassengerMaxPoolSize to 200 instead of 12 (yes…). > > > > I have found out that enabling/disabling the SSH keys collection on a > specific host causes the compilation time to jump from 17s to 53s or even > more. > > > > I have tried querying the puppetdb for those specific Sshkey resources > using curl and to my surprise, this was quite quick. > > I am currently profiling the puppet master and will send that data to > puppetlabs if that’s still usefull to them (see : > https://puppetlabs.com/blog/tune-puppet-performance-profiler), but I > would like to know if someone would understand what’s wrong with that use > of exported resources ? > > > > I’m not putting tons of things in there, so I can’t understand how this > can have such a huge impact on my “relatively” sized cluster.. > > Any known workaround other than adding more puppet masters ? (I know it > might be possible to query the puppetdb directly from a template, but I do > not own the saz/ssh module, so that would be some work to create a pull > request just to work around… well, a “basic” feature) > > My last resort would be to disable the use of exported resources for the > ssh keys and revert to my hack, but I can’t do that for my monitoring > anyway, and that would surely also benefit from a fix… > > > > I don’t know if I could use the puppet server (not master) easily with > foreman, maybe that’d be something to try too… > > > > Thanks for any hint && Regards ! > > Frederic Schaer > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA231747172E%40E-EXDAGE-A0.extra.cea.fr > <https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA231747172E%40E-EXDAGE-A0.extra.cea.fr?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAK4ABDJyd31bEMg%3DqLmS7pW0zezWP%3Du2nwu7wgXmxzW8RpnLMA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
