On 04/07/14 02:16 PM, Giuseppe Ragusa wrote:
Hi all,
I'm trying to create a script as per subject (on CentOS 6.5,
CMAN+Pacemaker, only DRBD+KVM active/passive resources; SNMP-UPS
monitored by NUT).
Ideally I think that each node should stop (disable) all locally-running
VirtualDomain resources (doing so cleanly demotes than downs the DRBD
resources underneath), then put itself in standby and finally shutdown.
On further startup, manual intervention would be required to unstandby
all nodes and enable resources (nodes already in standby and resources
already disabled before blackout should be manually distinguished).
Is this strategy conceptually safe?
Unfortunately, various searches have turned out no "prior art" :)
I started work on something similar with apcupsd (first I had to make it
work with multiple UPSes, which I did). Then I decided not to actually
implement, and decided instead to leave it up to an admin to decide
how/when/if to initiate a graceful shutdown.
My rationale was that this placed way too much potential damage in the
hands of, effectively, a single trigger. One bad bug and you could bring
down a perfectly fine cluster.
Instead, what I did was ensure that any power event triggered an alert
email (x2, as both nodes ran the monitoring app). This way, I (and the
client's admins) would be notified immediately if anything happened.
Then it was up to us to decide how/if to initiate a graceful shutdown.
One real-world example;
A couple months ago, a client's neighborhood was hit with a prolonged
power outage. Eventually, we decided to gracefully shut down. However,
one of the windows VMs had downloaded and prepped to install about 30
updates (no idea how this happened, except windows). Anyway, the VM took
more time to shut down than the batteries could support. So half-way
through, we withdrew one node and powered it off to shed load and gain
battery runtime. This kind of logic can not reasonably be coded into a
script.
My $0.02.
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
