On Tue, Jul 01, 2008 at 04:04:54PM +0200, [EMAIL PROTECTED] wrote: > After I had successfully upgraded this cluster to the new OS I was > wondering, why my Nagios plugin always returned CRITICAL states > though heartbeat was running on the node at the time. > Then I discovered that the output of my check command differed > decisively depending on who executed the check. > > e.g. as root I get > > # /usr/lib64/nagios/plugins/custom/check_heartbeat.sh > OK - heartbeat is running on nodeA > > or rather what really gets executed in that plugin and whose > output merely gets parsed is > > # /usr/lib64/heartbeat/heartbeat -s > heartbeat OK [pid 31017 et al] is running on nodeA [nodeA]... > > # pgrep -P1 -fl heartbeat > 31017 heartbeat: master control process > > But when run as an unprivileged user, as is the case when the nrpe > daemon is executing the check, oops, I get this strange result > > # /usr/lib64/nagios/plugins/check_nrpe -n -H localhost -c check_heartbeat > CRITICAL - heartbeat is stopped on nodeA > > How come, is this a bug or intended behavior?
I've just had a quick look through the source to see what the -s flag actually does (I'll need to set up monitoring of heartbeat in Nagios shortly, as it happens). It reads the PID file and then checks if the process is running, and that the process with the PID it's checking is actually heartbeat (by checking that its /proc/.../exe is a link to the heartbeat binary). On my system, even though the process directory and the symlinks therein appear to be world-readable, they're not: $ ls -la /proc/`sed 's/ *//' /var/run/heartbeat.pid` ls: cannot read symbolic link /proc/18467/cwd: Permission denied ls: cannot read symbolic link /proc/18467/root: Permission denied ls: cannot read symbolic link /proc/18467/exe: Permission denied When heartbeat tries to ascertain that the process running with that particularly pid is actually heartbeat, it encounters an error and therefore fails. I'm not sure if this aspect of the proc filesystem's behaviour can be adjusted, or if it's desirable to adjust it. So, I would suggest one of: 1. Go with your approach of just checking the process listing 2. Set up sudo or similar so Nagios can do the check 3. Set up a scheduled job to do a check as root, and write the result status code and a line of output to a file somewhere. Then the Nagios check command can check that the status file was updated recently, and if so use that for its own response. I'll probably go with option #2 or #3, but I haven't really looked into how exactly I'm going to ascertain that heartbeat is up and running. Possibly I'll use crm_mon -1 and check that the expected nodes are both online, and set a warning status if either is offline (and critical if I can't work out their status at all). _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
