On Tue, 2010-12-14 at 00:24 -0800, Chris wrote: > Hi > > I recently upgraded my puppet masters (and clients) from 0.24.8 to > 2.6.4 > > Previously, my most busy puppet master would hover around about 0.9 > load average, after the upgrade, its load hovers around 5 > > I am running passenger and mysql based stored configs. > > Checking my running processes, ruby (puppetmasterd) shoots up to 99% > cpu load and stays there for a few seconds before dropping again. > Often there are 4 of these running simultaneously, pegging each core > at 99% cpu.
I would say it is perfectly normal. Compiling the catalog is a hard and complex problem and requires CPU. The difference between 0.24.8 and 2.6 (or 0.25 for what matters) is that some performance issues have been fixed. Those issues made the master be more I/O bound under 0.24, but now mostly CPU bound in later versions. Now compare the compilation time under 0.24.8 and 2.6 and you should see that it reduced drastically (allowing to fit more compilation in the same time unit). The reverse of the medal is that now your master requires transient high CPU usage. I don't really get what is the issue about using 100% of CPU? You're paying about the same price when your CPU is used and when it's idle, so that shouldn't make a difference :) If that's an issue, reduce the concurrency of your setup (run less compilation in parallel, implement splay time, etc...). > It seems that there has been a serious performance regression between > 0.24 and 2.6 for my configuration I think it's the reverse that happened. > I hop the following can help work out where... > > I ran puppetmasterd through a profiler to find the root cause of this > (http://boojum.homelinux.org/profile.svg). The main problem appears > to be in /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/resource.rb, in > the evaluate function. > > I added a few timing commands around various sections of that function > to find the following breakdown of times spent inside it, and the two > most intensive calls are > --- > paramobjects = parameters.collect { |param| > param.safeevaluate(scope) > } > --- > > and > --- > resource_titles.flatten.collect { |resource_title| > exceptwrap :type => Puppet::ParseError do > resource = Puppet::Parser::Resource.new( > fully_qualified_type, resource_title, > :parameters => paramobjects, > :file => self.file, > :line => self.line, > :exported => self.exported, > :virtual => virt, > :source => scope.source, > :scope => scope, > :strict => true > ) > > if resource.resource_type.is_a? Puppet::Resource::Type > resource.resource_type.instantiate_resource(scope, resource) > end > scope.compiler.add_resource(scope, resource) > scope.compiler.evaluate_classes([resource_title],scope,false) > if fully_qualified_type == 'class' > resource > end > }.reject { |resource| resource.nil? } > --- Yes, this is what the compiler is doing during compilation: evaluating resources and parameters. The more resources you use, the more the compilation will take time and CPU. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
