On 15/12/10 20:24, Disconnect wrote: > On Wed, Dec 15, 2010 at 2:14 PM, Brice Figureau > <brice-pup...@daysofwonder.com <mailto:brice-pup...@daysofwonder.com>> > wrote: >> >> Note: we were talking about the puppet master taking 100% CPU, but >> you're apparently looking to the puppet agent, which is a different story. >> > > The agent isn't taking cpu, it is hanging waiting for the master to do > anything. (The run I quoted earlier eventually ended with a timeout..) > The master has pegged the cpus, and it seems to be related to file > resources:
Oh, I see. > $ ps auxw|grep master > puppet 31392 74.4 4.7 361720 244348 ? R 10:42 162:06 Rack: > /usr/share/puppet/rack/puppetmasterd > > > puppet 31396 70.0 4.9 369524 250200 ? R 10:42 152:32 Rack: > /usr/share/puppet/rack/puppetmasterd > > > puppet 31398 66.2 3.9 318828 199472 ? R 10:42 144:10 Rack: > /usr/share/puppet/rack/puppetmasterd > > > puppet 31400 66.6 4.9 369992 250588 ? R 10:42 145:04 Rack: > /usr/share/puppet/rack/puppetmasterd > > > puppet 31406 68.6 3.9 318292 200992 ? R 10:42 149:31 Rack: > /usr/share/puppet/rack/puppetmasterd > > > puppet 31414 67.0 2.4 243800 124476 ? R 10:42 146:00 Rack: > /usr/share/puppet/rack/puppetmasterd > Note that they're all running. That means there is none left to serve file content if they are all busy for several seconds (in our case around 20) while compiling catalogs. > Dec 15 13:42:23 puppet puppet-master[31406]: Compiled catalog for > puppet.foo.com <http://puppet.foo.com> in environment production in > 30.83 seconds > Dec 15 13:42:49 puppet puppet-agent[10515]: Caching catalog for > puppet.foo.com <http://puppet.foo.com> > Dec 15 14:00:18 puppet puppet-agent[10515]: Applying configuration > version '1292438512' > ... > Dec 15 14:14:56 puppet puppet-agent[10515]: Finished catalog run in > 882.43 seconds > Changes: > Total: 6 > Events: > Success: 6 > Total: 6 > Resources: > Changed: 6 > Out of sync: 6 > Total: 287 That's not a big number. > Time: > Config retrieval: 72.20 This is also suspect. > Cron: 0.05 > Exec: 32.42 > File: 752.33 Indeed. > Filebucket: 0.00 > Mount: 0.98 > Package: 6.13 > Schedule: 0.02 > Service: 9.09 > Ssh authorized key: 0.07 > Sysctl: 0.00 > > real 34m56.066s > user 1m6.030s > sys 0m26.590s > That just means your master are so busy serving catalogs that they barely have the time to serve files. One possibility is to use file content offloading (see one of my blog post about this: http://www.masterzen.fr/2010/03/21/more-puppet-offloading/). How many nodes are you compiling at the same time? Apparently you have 6 master processes running at high CPU usage. As I said earlier, I really advise people to try puppet-load (which can be found in the ext/ directory of the source tarball since puppet 2.6) to execise load againts a master. This will help you find your actual concurrency. But, if it's a bug, then could this be an issue with passenger? -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
