I am replying to my own question because I want to share this
knowledge of how I resolved this issue.
My objective was to get almost every aspect of system under puppet
control. So I started by installing a very bare Slackware (few
packages, just enough to get machine up and running with ssh. Then I
started modifications of the system by means of puppet. I built a nice
set of modules which handled the basics, like system config files and
libraries and various tool installations. Then I wrapped all this in a
single class, like this:
class groundwork {
include 'sys'
include 'openssl'
include 'libs'
include 'tools'
include 'ssh'
}
Then I started adding real stuff, production services, like httpd,
like this:
class httpd {
file {file1:...}
file {file2: ...}
package { 'httpd': ...}
file { '/etc/rc.d/rc.httpd': ...}
file { '/var/www': ...}
service {'httpd': ...}
}
And there were consequential 'require' relationships to these items
above and the first one included this line:
file {'file1':
require => Class['groundwork'],
}
And I thought that this line meant a dependency like this:
file1
|
| (depends on)
|
groundwork(class)
|
| (which in turn depends on)
|
ssh (class)
|
| (which in turn depends on)
|
file {'/etc/rc.d/rc.sshd':...}
file {'/etc/ssh/sshd_config':...}
package {'openssh': ...}
etc.
But what it REALLY means is that my item (in this case, file1) was
dependent on every real item in the groundwork class and all the
classes included. Every file from ssh class, and every file from libs
class and every file from SYS class.
This dependency tree grew so large that puppet was unable to walk it
in decent time. Ergo, lesson learned: puppet maintains file
dependencies by itself, so you should only include those dependencies
that your item in question really depends on, not everything that YOU
THINK should be done before.
I hope this information helps someone to avoid spending couple of days
of debugging. And relationship graphs (from relationships.dot and
expanded_relationships.dot) really helped resolving this issue.
Best regards,
b.
On 26 sep., 00:23, Bostjan Skufca <bostjan.sku...@gmail.com> wrote:
> Hi again,
>
> puppetd started to hang with100% CPU usage all of the sudden. Strace
> just keeps repeating these 2 lines:
>
> --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
> sigreturn() = ? (mask now [])
>
> In logs it hangs with this message:
> debug: Calling fileserver.describe
>
> But lots of these fileserver.describe calls prior to hang are
> successful.
>
> If I run it with puppetd --test --debug and hit Ctrl+C when it is
> "hanged", I get this trace:
> debug: Signal caught here:
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:28:in `call'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:28:in `adjacent'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:232:in `adjacent'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:262:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `each'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> simple_graph.rb:260:in `walk'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> pgraph.rb:116:in `tree_from_vertex'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> pgraph.rb:37:in `dependencies'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:346:in `failed_dependencies?'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:619:in `skip?'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:219:in `eval_resource'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:310:in `evaluate'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> util.rb:425:in `thinmark'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/1.8/benchmark.rb:308:in
> `realtime'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> util.rb:424:in `thinmark'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:309:in `evaluate'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:303:in `collect'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> transaction.rb:303:in `evaluate'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/node/
> catalog.rb:124:in `apply'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> network/client/master.rb:256:in `run'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> util.rb:180:in `benchmark'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/1.8/benchmark.rb:308:in
> `realtime'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> util.rb:179:in `benchmark'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> network/client/master.rb:255:in `run'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/1.8/sync.rb:229:in
> `synchronize'
> debug: /usr/local/puppet-0.24.8-2/lib/ruby/site_ruby/1.8/puppet/
> network/client/master.rb:237:in `run'
> debug: /usr/local/puppet/bin/puppetd:417
> notice: Shutting down
>
> Are there any common diagnoses for these simptoms? Cyclic reference or
> something?
>
> Thanks,
> Bostjan
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
