On Oct 5, 4:36 am, Scott Smith <sc...@ohlol.net> wrote:
> It doesn't matter if puppetmasterd or puppetd are running and working if
> your clients are failing catalog runs.
>
> Send reports, write a check that alerts on N hosts with failed reports over
> X timeframe or something.

I favor monitoring each node rather than try to figure out if the
master works
or not as this is ultimately the outcome you care for - can my nodes
do puppet runs.

Puppet now saves its last run status in the libdir:

/var/lib/puppet/state/last_run_summary.yaml:
---
  time:
    group: 0.001692
    last_run: 1317804488
    class: 0.003929
    yumrepo: 0.020103
    service: 9.017434
    schedule: 0.004151
    cron: 0.010546
    config_retrieval: 15.0572321414948
    total: 34.9742621414947
    package: 0.588751
    filebucket: 0.000687
    file: 8.895422
    exec: 1.361625
    user: 0.01269
  resources:
    total: 414
  events:
    total: 0
  changes:
    total: 0

I wrote a nagios check to check this and report on any problems:

When last did puppet run:

# check_puppet.rb -c 3600 -w 1800
OK: Puppet is currently enabled, last run 712 seconds ago with 0
failures

how many resources failed:
# check_puppet.rb -c 5 -w 2 --check-failures
OK: Puppet is currently enabled, last run 742 seconds ago with 0
failures

and you can also add --only-enabled to have machines undergoing
maintenance not show
up as critical.

I run this under nrpe etc.

The last run summary yaml file is a bit odd in that it doesn't always
contain all the information
for instance a syntax error in catalogs will have it only include 1
piece of data and other differences
during execution, hopefully this will improve in time (I should
probably file tickets)

The code for the check is @ 
https://github.com/ripienaar/monitoring-scripts/blob/master/puppet/check_puppet.rb
and it is compatible with all the current odd behaviors of the file as
of 2.6.9

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Reply via email to