Full automation is planned but does not eliminate the need for the software.
Zero human auditing of fully automated processes and data collection are not
acceptable to various certifying entities, the relevant auditors, the
inevitably involved lawyers, and won’t pick up on bad data, like a bad
thermometer or snmp counter that says a CRAC is 65 degrees when it’s really 90.
So I’m still going to need a management solution to the issue whether it’s to
tell someone to do the work or to tell someone to check the automated work.
David
On 7/27/16, 7:19 PM, "Lee" <ler...@gmail.com> wrote:
On 7/27/16, David Hubbard <dhubb...@dino.hostasaurus.com> wrote:
> Hi all, curious if anyone has recommendations on software that helps
manage
> routine duties assigned to operations staff?
Have computers do the routine scut work - not people.
> For example, let’s say we have a P&P that says someone from the netops
group
> must check that Rancid is successfully backing up all router configs
> bi-weekly.
You've got the source code for rancid, so change rancid-run to do something
like
LOGFILE=$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S`; export LOGFILE
change the
) >$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S` 2>&1
to
) >$LOGFILE 2>&1
and then in control_rancid do something like
grep "clogin error:" $LOGFILE | sort | uniq -c >$TMP.fail
if [ -s $TMP.fail ]; then
# got some output, mail the report
...
Do the same type thing for checking on
> backup failures, backup internet circuit status, out of band interfaces,
etc.
Automate the checks, put the scripts in crontab & mail out an
"OhNoes!" or "all clear" msg at the end. At which point you're left
with the problem of making sure the managers are looking at the emails
& making sure whatever problems are found actually get fixed :)
Regards,
Lee
