As attachments, then. Karl
On Tue, Sep 22, 2009 at 10:51 AM, Dejan Muhamedagic <[email protected]>wrote: > Hi, > > On Tue, Sep 22, 2009 at 09:39:00AM -0400, Karl W. Lewis wrote: > > On Mon, Sep 7, 2009 at 11:23 AM, Karl W. Lewis <[email protected] > >wrote: > > > > > > > > > > > On Mon, Sep 7, 2009 at 10:16 AM, Dejan Muhamedagic < > [email protected]>wrote: > > > > > >> Hi, > > >> > > >> On Wed, Aug 26, 2009 at 01:11:24PM -0400, Karl W. Lewis wrote: > > >> > On Wed, Aug 26, 2009 at 10:58 AM, Dejan Muhamedagic < > > >> [email protected]>wrote: > > >> > > > >> > > Hi, > > >> > > > > >> > > On Mon, Aug 24, 2009 at 11:35:15AM -0400, Karl W. Lewis wrote: > > >> > > > I don't know if anyone else is running Heartbeat/Pacemaker on an > > >> Egenera > > >> > > > BladeFrame, but if they are, these scripts might be useful to > help > > >> set up > > >> > > > stonith. > > >> > > > > > >> > > > The Egeneras use an IPMI arrangement to control the various > > >> processor > > >> > > > blades, and these routines take advantage of that. > > >> > > > > >> > > Did you try ipmilan or external/ipmi too? > > >> > > > >> > > > >> > I did not, but a quick review leads me to believe that the egenera > frame > > >> > does not count as an impi "device". One might be able to make that > > >> work, > > >> > but you'd have to modify the ipmi script until it looked much like > this > > >> > example, or, you'd have to play very. very hard with the egnera > > >> c-blades, > > >> > and I can't imagine anyone would want to do that. > > >> > > >> Well, either they are IPMI compliant or not, i.e. our ipmi > > >> stonith modules should work without modifications. I think it's > > >> worth trying. At any rate, I think that this is a different > > >> solution worth pursuing, since it would reduce the configuration > > >> quite a bit. > > >> > > > > > > I've not forgotten this task, but it's been very busy at work and I'm > doing > > > more research. The ipmi commands do not work the way I expected them > to, > > > (writing to "off" just resets the blade), so I'm looking at the native > > > Egenera PAN Manager commands. > > > > > >> > > >> > > > This may not be the best method of doing it, but it does appear > to > > >> work. > > >> > > > It is not fast. What it lacks in speed, however, it makes up > for in > > >> > > > flexibility. It can figure out which c-blade is the master, and > > >> then > > >> > > figure > > >> > > > out which p-blade is running the server to be stonithed, and > then > > >> > > stonith, > > >> > > > it. The framework for this was an existing external stonith > script > > >> that > > >> > > I > > >> > > > simply modified to meet my needs. > > >> > > > > > >> > > > The two short shell scripts at the top need to live in > /root/bin/ on > > >> both > > >> > > > > >> > > I guess you wanted to say $HOME/bin. Note that the root user may > > >> > > not always have the same home directory. Perhaps it would be > > >> > > better to go with an absolute path such as /usr/local/bin. > > >> > > > >> > > > >> > I actually put them in root's directory with malice aforethought, > but > > >> now > > >> > that you've asked the question... I guess I can't defend that > decision. > > >> I > > >> > use the user root because that's the only user on the c-blades under > > >> > egenera's setup. One could add another user, but I can't advocate > > >> changing > > >> > the config of the c-blades. > > >> > > > >> > > > >> > > > > >> > > > > >> > > > control blades. The cluster servers need to have ssh keys and > any > > >> > > required > > >> > > > firewall rules setup such that root on the cluster servers can > ssh > > >> to the > > >> > > > c-blades. > > >> > > > > >> > > There can be more than one? Is that covered in any way by the > > >> > > script, i.e. does it try all available? > > >> > > > >> > > > >> > Yes. Just so. There are two c-baldes; one is master and one is > > >> standby. I > > >> > *think* that the master is the only one that can issue the ipmi > commands > > >> and > > >> > get a reaction. I've not experimented with that, however. (I hit a > > >> > solution that worked, and I stopped experimenting.) > > >> > > >> It would be good if you do some :) You don't want any surprises > > >> when it comes to fencing. > > >> > > > > > > Right. Surprises are for Birthday Parties, presents, and such, not so > much > > > for servers. > > > > > > Karl > > > > > > > > > > > >> > > >> > The script tracks down > > >> > which of the two is master and then talks to that server. One > could > > >> avoid > > >> > that by giving both c-blades the same set of rsa host keys, and then > > >> just > > >> > talking to the virtual IP Address that the master blade, which ever > of > > >> the > > >> > two it is, is answering on. Again, I am disinclined to modify the > > >> egenera > > >> > setup that way. I tried telling ssh to ignore the key verification, > but > > >> it > > >> > still updates the knownhosts file when the script logs in, so it'll > > >> throw > > >> > errors the next time it tries if the master has moved. > > >> > > >> Right. Isn't there an option to tell ssh not to update the > > >> knownhosts file. > > >> > > >> > > > If anyone else is already doing this and has a better way, I'd > be > > >> happy > > >> > > to > > >> > > > hear about it. > > >> > > > > > >> > > > Karl > > >> > > > > >> > > > ### THESE TWO SHELL SCRIPTS ARE EXPECTED IN /root/bin ON BOTH > > >> c-blades > > >> > > > ### THAT WAY THE SCRPIT THAT FOLLOWS WILL WORK REGARDLESS OF > WHICH > > >> > > > ### EGENERA c-blade IS PAN-MASTER > > >> > > > > > >> > > > #!/bin/bash > > >> > > > # get_master.sh - returns the name of the master control blade > in > > >> the > > >> > > > PAN. > > >> > > > > > >> > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c > 2` > > >> > > > echo "cblade-${MASTER}" > > >> > > > #CUT HERE > > >> ############################################################# > > >> > > > > > >> > > > #!/bin/bash > > >> > > > > > >> > > > # get_processor_blade.sh - identifies the blade running a given > > >> pserver. > > >> > > > > >> > > This name doesn't exactly match the references in the script. > > >> > > > >> > > > >> > > > >> > :%s/_blade/-blade/ > > >> > > > >> > I'm a lousy typist.... > > >> > > > >> > > > >> > > > >> > > > > >> > > > > >> > > > BLADE=`/opt/panmgr/bin/esh pan | grep ^${1}` > > >> > > > echo ${BLADE} | sed -e "s/^.*MyBladeFrame\/\(.\)\(.\) > .*/a\1-\2/" > > >> > > > #CUT HERE > > >> ############################################################# > > >> > > > > > >> > > > #!/bin/bash > > >> > > > # > > >> > > > # External STONITH module for Egenera. > > >> > > > # > > >> > > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree < > > >> [email protected]> > > >> > > > # Copyright (c) 2009 Karl W. Lewis <[email protected]> > > >> > > > # This program is free software; you can redistribute > > >> > > > #it and/or modify # it under the terms of version 2 of the GNU > > >> General > > >> > > > #Public License as # published by the Free Software Foundation. > > >> > > > # > > >> > > > # This program is distributed in the hope that it would be > useful, > > >> but > > >> > > > # WITHOUT ANY WARRANTY; without even the implied warranty of > > >> > > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > >> > > > # > > >> > > > # Further, this software is distributed without any warranty > that it > > >> is > > >> > > > # free of the rightful claim of any third person regarding > > >> infringement > > >> > > > # or the like. Any license provided herein, whether implied or > > >> > > > # otherwise, applies only to this software file. Patent > licenses, > > >> if > > >> > > > # any, provided herein do not apply to combinations of this > program > > >> with > > >> > > > # other software, or any other product whatsoever. > > >> > > > # > > >> > > > # You should have received a copy of the GNU General Public > License > > >> > > > # along with this program; if not, write the Free Software > > >> Foundation, > > >> > > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. > > >> > > > # > > >> > > > > > >> > > > SSH_COMMAND="/usr/bin/ssh" > > >> > > > > > >> > > > #LOG=/var/log/egenera-stonith.log > > >> > > > #echo `date` >> $LOG > > >> > > > #echo $hostlist >> $LOG > > >> > > > #echo $1 >> $LOG > > >> > > > #echo $2 >> $LOG > > >> > > > #echo "executing--------------" >> $LOG > > >> > > > > > >> > > > # Rewrite the hostlist to accept "," as a delimeter for > hostnames > > >> too. > > >> > > > hostlist=`echo $hostlist | tr ',' ' '` > > >> > > > > > >> > > > is_host_up() { > > >> > > > for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 > > >> > > > do > > >> > > > if > > >> > > > ping -w1 -c1 "$1" >/dev/null 2>&1 > > >> > > > then > > >> > > > sleep 10 > > >> > > > else > > >> > > > return 1 > > >> > > > fi > > >> > > > done > > >> > > > return 0 > > >> > > > } > > >> > > > > >> > > This is from external/ssh and used only if the parameter > > >> > > livedangerously is set to yes. Doesn't strike me as a very robust > > >> > > way to check if a node is running. Is there another way? Using > > >> > > the cblades sounds like a most natural way. > > >> > > > >> > > > >> > Yes. You're clearly right. By way of excuse I'd claim I was rather > > >> > narrowly focused on getting a server fenced and therefore I ignored > > >> other > > >> > functions. If you'll tolerate a "version 2, (or, I suppose v0.02), > I'll > > >> try > > >> > again. It'll take me a few hours to fix and test, I'll try again. > (A > > >> quick > > >> > experiment reveals that yes, the master blade does indeed know the > > >> status of > > >> > a given processor blade.) > > >> > > >> Great. > > >> > > >> > > > MASTER_BLADE=`$SSH_COMMAND cblade-1 "bin/get_master.sh" | tail > -1` > > >> > > > > > >> > > > case $1 in > > >> > > > gethosts) > > >> > > > for h in $hostlist ; do > > >> > > > echo $h > > >> > > > done > > >> > > > exit 0 > > >> > > > ;; > > >> > > > on) > > >> > > > for h in $hostlist > > >> > > > do > > >> > > > if > > >> > > > [ "$h" != "$2" ] > > >> > > > then > > >> > > > continue > > >> > > > else > > >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > >> > > "bin/get_processor-blade.sh $h"` > > >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/on" > > >> > > > fi > > >> > > > sleep 30 > > >> > > > if > > >> > > > is_host_up $h > > >> > > > then > > >> > > > exit 1 > > >> > > > else > > >> > > > exit 0 > > >> > > > fi > > >> > > > done > > >> > > > exit 0 > > >> > > > ;; > > >> > > > off) > > >> > > > for h in $hostlist > > >> > > > do > > >> > > > if > > >> > > > [ "$h" != "$2" ] > > >> > > > then > > >> > > > continue > > >> > > > else > > >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > >> > > "bin/get_processor-blade.sh $h"` > > >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/off" > > >> > > > fi > > >> > > > done > > >> > > > exit 0 > > >> > > > ;; > > >> > > > reset) > > >> > > > for h in $hostlist > > >> > > > do > > >> > > > if > > >> > > > [ "$h" != "$2" ] > > >> > > > then > > >> > > > continue > > >> > > > else > > >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE > > >> > > "bin/get_processor-blade.sh $h"` > > >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > > > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/nmi" > > >> > > > fi > > >> > > > sleep 20 > > >> > > > if > > >> > > > is_host_up $h > > >> > > > then > > >> > > > exit 1 > > >> > > > else > > >> > > > exit 0 > > >> > > > fi > > >> > > > done > > >> > > > exit 1 > > >> > > > ;; > > >> > > > > >> > > on), off), and reset) cases are very similar, they should be made > > >> > > to share some code. Also, the sleep values look too ephemeral for > > >> > > my taste. There should be a relatively tight loop to check the > > >> > > host status. > > >> > > > > >> > > > >> > I'm not sure yet what to do about the "ephemeral" sleep values. I > chose > > >> > arbitrarily long waits because egenera blades take a surprisingly > long > > >> time > > >> > to boot. Your point is well taken, however, the code can be > improved. > > >> > > >> Looking forward to it :) > > >> > > >> > > > status) > > >> > > > if > > >> > > > [ -z "$hostlist" ] > > >> > > > then > > >> > > > exit 1 > > >> > > > fi > > >> > > > for h in $hostlist > > >> > > > do > > >> > > > if > > >> > > > ping -w1 -c1 "$h" 2>&1 | grep "unknown host" > > >> > > > > >> > > The status should return the status of the device, not the status > > >> > > of a node. > > >> > > > >> > > > >> > ?? I must have misunderstood the existing code. Hmmmmm. The > status of > > >> > this device is always "0" if there's a hostlist and if $MASTER_BLADE > is > > >> set, > > >> > I think. > > >> > > >> And if the c-blade is reachable. Something like > > >> "ssh $MASTER_BLADE true" should do. > > >> > > >> > Thank you for taking the itme to look this over. I'll clean it up > some > > >> and > > >> > try again. > > >> > > >> Many thanks. > > >> > > >> Cheers, > > >> > > >> Dejan > > >> > > >> > Be well, > > >> > > > >> > Karl > > >> > > > >> > > > > >> > > > > >> > > Cheers, > > >> > > > > >> > > Dejan > > >> > > > > >> > > > > > > > > I've cleaned this up some, and reworked much of it. It has been tested > in > > production for a couple of weeks now, and it works properly as near as I > can > > tell. (It has properly restarted a misbehaving server.) > > > > I could not see a way to get the existing IPMI script to work with > Egenera, > > and I have abandoned using anything other than native Egenera commands > to > > control the blades. These scripts require that ssh keys are set up for > user > > root between the cluster servers and the c-blades. > > OK. > > > Code: > > Can you please send the scripts as attachments. The mail agents > kill the formatting. > > > #The two bash scripts here at the top go on each of the Egenera C-Blades. > > #the STONITH module goes in /usr/lib/stonith/plugins/external/egenera on > > #each of the servers in the cluster. > > These two scripts will go to the doc part of the package. > > Cheers, > > Dejan > > > ############################### cut here > > ###################################### > > #!/bin/bash > > # get_status.sh - returns the status of a pServer, given the LPAN and > > pServer > > > > #called as: > > # get_status.sh pServer LPAN_Name UP > > # or > > # get_status.sh pServer LPAN_Name DOWN > > #depending on what you want. This only works of the pServer is running > the > > PAN Agent > > #because until the agent comes up, the server does not count as "Up" > > > > if [ ${3} = "UP" ] > > then > > CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} > | > > grep "Booted" | grep "Up"` > > if [ -z "${CONDITION}" ] > > then > > echo "Down" # No Joy > > else > > echo "Up" # Joy! > > fi > > else # it should be the case that ${3} = "DOWN" > > CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} > | > > grep "Shutdown"` > > if [ -z "${CONDITION}" ] > > then > > echo "Up" # No Joy > > else > > echo "Down" # Joy! > > fi > > fi > > > > ############################### cut here > > ###################################### > > > > #!/bin/bash > > # get_master.sh - returns the name of the master control blade in the > PAN. > > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2` > > echo "cblade-${MASTER}" > > > > ############################### cut here > > ###################################### > > > > #!/bin/bash > > # > > # External STONITH module for Egenera pServers. > > # > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <[email protected]> > > # Copyright (c) 2009 Karl W. Lewis <karl.w.lewis - AT - gmail.com> > > # > > # This program is free software; you can redistribute it and/or modify > > # it under the terms of version 2 of the GNU General Public License as > > # published by the Free Software Foundation. > > # > > # This program is distributed in the hope that it would be useful, but > > # WITHOUT ANY WARRANTY; without even the implied warranty of > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > # > > # Further, this software is distributed without any warranty that it is > > # free of the rightful claim of any third person regarding infringement > > # or the like. Any license provided herein, whether implied or > > # otherwise, applies only to this software file. Patent licenses, if > > # any, provided herein do not apply to combinations of this program with > > # other software, or any other product whatsoever. > > # > > # You should have received a copy of the GNU General Public License > > # along with this program; if not, write the Free Software Foundation, > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. > > # > > > > SSH_COMMAND="/usr/bin/ssh -q -x -n -l root" > > GET_MASTER="/root/bin/get_master.sh" > > GET_STATUS="/root/bin/get_status.sh" > > PSERVER_COMMAND="/opt/panmgr/bin/esh pserver" > > > > #These TIMEOUT values were obtained from the 'pserver -l pan/pserver' > > command > > # on a given Egenera frame under "Timeouts" YMMV. > > > > BOOT_TIMEOUT=270 > > REBOOT_TIMEOUT=450 > > SHUTDOWN_TIMEOUT=180 > > > > TIMEOUT=${REBOOT_TIMEOUT} > > STATUS="UNKNOWN" > > > > #LOG=/var/log/egenera-stonith.log > > #echo `date` >> ${LOG} > > #echo ${hostlist} >> ${LOG} > > #echo ${lpan_name} >> ${LOG} > > #echo $1 >> ${LOG} > > #echo $2 >> ${LOG} > > #echo "executing--------------" >> ${LOG} > > > > # Rewrite the hostlist to accept "," as a delimeter for hostnames too. > > hostlist=`echo ${hostlist} | tr ',' ' '` > > > > is_host_up() { > > TIME=$((${TIMEOUT}/5)) > > for j in 1 2 3 4 5 6 > > do > > STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h} > > ${lpan_name} UP"` > > if [ "${STATUS}" = "Down" ] > > then > > sleep ${TIME} > > else > > SYSTEM_STATUS="On Target" > > return 1 > > fi > > done > > return 0 > > } > > > > is_host_down() { > > TIME=$((${TIMEOUT}/5)) > > for j in 1 2 3 4 5 6 > > do > > STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h} > > ${lpan_name} DOWN"` > > if [ "${STATUS}" = "Up" ] > > then > > sleep ${TIME} > > else > > SYSTEM_STATUS="On Target" > > return 1 > > fi > > done > > return 0 > > } > > > > MASTER_BLADE=`${SSH_COMMAND} cblade-1 "${GET_MASTER}" | tail -1` > > > > if [ -z "${MASTER_BLADE}" ] > > then > > MASTER_BLADE=`${SSH_COMMAND} cblade-2 "${GET_MASTER}" | tail -1` > > fi > > > > case ${1} in > > gethosts) > > for h in ${hostlist} > > do > > echo ${h} > > done > > exit 0 > > ;; > > on|off|reboot) > > for h in ${hostlist} > > do > > if [ "${h}" != "${2}" ] > > then > > continue > > else > > if [ "${1}" = "on" ] > > then > > CMD="--boot" > > elif [ "${1}" = "off" ] > > then > > CMD="--shutdown --force" > > else #must be "reboot" > > CMD="--reboot --force" > > fi > > ${SSH_COMMAND} ${MASTER_BLADE} "${PSERVER_COMMAND} ${CMD} > > ${lpan_name}/${h}" > > if [ "${1}" = "on" ] > > then > > TIMEOUT=${BOOT_TIMEOUT} > > elif [ "${1}" = "reboot" ] > > then > > TIMEOUT=${REBOOT_TIMEOUT} > > else > > TIMEOUT=${SHUTDOWN_TIMEOUT} > > fi > > if [ "${1}" = "off" ] > > then > > is_host_down ${h} > > else > > is_host_up ${h} > > fi > > fi > > if [ "${SYSTEM_STATUS}" = "On Target" ] > > then > > exit 0 > > else > > exit 1 > > fi > > done > > exit 1 # We got here only if the host was not in the list.... > > ;; > > status) > > if [ -z "${hostlist}" ] || [ -z "${lpan_name}" ] || [ -z > > "${MASTER_BLADE}" ] > > then > > exit 1 > > fi > > exit 0 > > ;; > > getconfignames) > > echo "hostlist lpan_name" > > exit 0 > > ;; > > getinfo-devid) > > echo "Egenera STONITH device" > > exit 0 > > ;; > > getinfo-devname) > > echo "Egenera STONITH external device" > > exit 0 > > ;; > > getinfo-devdescr) > > echo "Egenera-based Linux host reset" > > exit 0 > > ;; > > getinfo-devurl) > > echo "http://www.egenera.com"; > > exit 0 > > ;; > > getinfo-xml) > > cat << EGENERAXML > > <parameters> > > <parameter name="hostlist" unique="1" required="1"> > > <content type="string" /> > > <shortdesc lang="en"> > > Hostlist > > </shortdesc> > > <longdesc lang="en"> > > The list of hosts that the STONITH device controls > > </longdesc> > > </parameter> > > <parameter name="lpan_name" unique="1" required="1"> > > <content type="string" /> > > <shortdesc lang="en"> > > LPAN_Name > > </shortdesc> > > <longdesc lang="en"> > > The name of the LPAN that the cluster blades, that this STONITH device > > controls, live in > > </longdesc> > > </parameter> > > </parameters> > > EGENERAXML > > exit 0 > > ;; > > *) > > exit 1 > > ;; > > esac > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems >
get_status.sh
Description: Bourne shell script
get_master.sh
Description: Bourne shell script
egenera
Description: Binary data
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
