On Mon, Sep 7, 2009 at 11:23 AM, Karl W. Lewis <[email protected]>wrote:
> > > On Mon, Sep 7, 2009 at 10:16 AM, Dejan Muhamedagic <[email protected]>wrote: > >> Hi, >> >> On Wed, Aug 26, 2009 at 01:11:24PM -0400, Karl W. Lewis wrote: >> > On Wed, Aug 26, 2009 at 10:58 AM, Dejan Muhamedagic < >> [email protected]>wrote: >> > >> > > Hi, >> > > >> > > On Mon, Aug 24, 2009 at 11:35:15AM -0400, Karl W. Lewis wrote: >> > > > I don't know if anyone else is running Heartbeat/Pacemaker on an >> Egenera >> > > > BladeFrame, but if they are, these scripts might be useful to help >> set up >> > > > stonith. >> > > > >> > > > The Egeneras use an IPMI arrangement to control the various >> processor >> > > > blades, and these routines take advantage of that. >> > > >> > > Did you try ipmilan or external/ipmi too? >> > >> > >> > I did not, but a quick review leads me to believe that the egenera frame >> > does not count as an impi "device". One might be able to make that >> work, >> > but you'd have to modify the ipmi script until it looked much like this >> > example, or, you'd have to play very. very hard with the egnera >> c-blades, >> > and I can't imagine anyone would want to do that. >> >> Well, either they are IPMI compliant or not, i.e. our ipmi >> stonith modules should work without modifications. I think it's >> worth trying. At any rate, I think that this is a different >> solution worth pursuing, since it would reduce the configuration >> quite a bit. >> > > I've not forgotten this task, but it's been very busy at work and I'm doing > more research. The ipmi commands do not work the way I expected them to, > (writing to "off" just resets the blade), so I'm looking at the native > Egenera PAN Manager commands. > >> >> > > > This may not be the best method of doing it, but it does appear to >> work. >> > > > It is not fast. What it lacks in speed, however, it makes up for in >> > > > flexibility. It can figure out which c-blade is the master, and >> then >> > > figure >> > > > out which p-blade is running the server to be stonithed, and then >> > > stonith, >> > > > it. The framework for this was an existing external stonith script >> that >> > > I >> > > > simply modified to meet my needs. >> > > > >> > > > The two short shell scripts at the top need to live in /root/bin/ on >> both >> > > >> > > I guess you wanted to say $HOME/bin. Note that the root user may >> > > not always have the same home directory. Perhaps it would be >> > > better to go with an absolute path such as /usr/local/bin. >> > >> > >> > I actually put them in root's directory with malice aforethought, but >> now >> > that you've asked the question... I guess I can't defend that decision. >> I >> > use the user root because that's the only user on the c-blades under >> > egenera's setup. One could add another user, but I can't advocate >> changing >> > the config of the c-blades. >> > >> > >> > > >> > > >> > > > control blades. The cluster servers need to have ssh keys and any >> > > required >> > > > firewall rules setup such that root on the cluster servers can ssh >> to the >> > > > c-blades. >> > > >> > > There can be more than one? Is that covered in any way by the >> > > script, i.e. does it try all available? >> > >> > >> > Yes. Just so. There are two c-baldes; one is master and one is >> standby. I >> > *think* that the master is the only one that can issue the ipmi commands >> and >> > get a reaction. I've not experimented with that, however. (I hit a >> > solution that worked, and I stopped experimenting.) >> >> It would be good if you do some :) You don't want any surprises >> when it comes to fencing. >> > > Right. Surprises are for Birthday Parties, presents, and such, not so much > for servers. > > Karl > > > >> >> > The script tracks down >> > which of the two is master and then talks to that server. One could >> avoid >> > that by giving both c-blades the same set of rsa host keys, and then >> just >> > talking to the virtual IP Address that the master blade, which ever of >> the >> > two it is, is answering on. Again, I am disinclined to modify the >> egenera >> > setup that way. I tried telling ssh to ignore the key verification, but >> it >> > still updates the knownhosts file when the script logs in, so it'll >> throw >> > errors the next time it tries if the master has moved. >> >> Right. Isn't there an option to tell ssh not to update the >> knownhosts file. >> >> > > > If anyone else is already doing this and has a better way, I'd be >> happy >> > > to >> > > > hear about it. >> > > > >> > > > Karl >> > > >> > > > ### THESE TWO SHELL SCRIPTS ARE EXPECTED IN /root/bin ON BOTH >> c-blades >> > > > ### THAT WAY THE SCRPIT THAT FOLLOWS WILL WORK REGARDLESS OF WHICH >> > > > ### EGENERA c-blade IS PAN-MASTER >> > > > >> > > > #!/bin/bash >> > > > # get_master.sh - returns the name of the master control blade in >> the >> > > > PAN. >> > > > >> > > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2` >> > > > echo "cblade-${MASTER}" >> > > > #CUT HERE >> ############################################################# >> > > > >> > > > #!/bin/bash >> > > > >> > > > # get_processor_blade.sh - identifies the blade running a given >> pserver. >> > > >> > > This name doesn't exactly match the references in the script. >> > >> > >> > >> > :%s/_blade/-blade/ >> > >> > I'm a lousy typist.... >> > >> > >> > >> > > >> > > >> > > > BLADE=`/opt/panmgr/bin/esh pan | grep ^${1}` >> > > > echo ${BLADE} | sed -e "s/^.*MyBladeFrame\/\(.\)\(.\) .*/a\1-\2/" >> > > > #CUT HERE >> ############################################################# >> > > > >> > > > #!/bin/bash >> > > > # >> > > > # External STONITH module for Egenera. >> > > > # >> > > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree < >> [email protected]> >> > > > # Copyright (c) 2009 Karl W. Lewis <[email protected]> >> > > > # This program is free software; you can redistribute >> > > > #it and/or modify # it under the terms of version 2 of the GNU >> General >> > > > #Public License as # published by the Free Software Foundation. >> > > > # >> > > > # This program is distributed in the hope that it would be useful, >> but >> > > > # WITHOUT ANY WARRANTY; without even the implied warranty of >> > > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. >> > > > # >> > > > # Further, this software is distributed without any warranty that it >> is >> > > > # free of the rightful claim of any third person regarding >> infringement >> > > > # or the like. Any license provided herein, whether implied or >> > > > # otherwise, applies only to this software file. Patent licenses, >> if >> > > > # any, provided herein do not apply to combinations of this program >> with >> > > > # other software, or any other product whatsoever. >> > > > # >> > > > # You should have received a copy of the GNU General Public License >> > > > # along with this program; if not, write the Free Software >> Foundation, >> > > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. >> > > > # >> > > > >> > > > SSH_COMMAND="/usr/bin/ssh" >> > > > >> > > > #LOG=/var/log/egenera-stonith.log >> > > > #echo `date` >> $LOG >> > > > #echo $hostlist >> $LOG >> > > > #echo $1 >> $LOG >> > > > #echo $2 >> $LOG >> > > > #echo "executing--------------" >> $LOG >> > > > >> > > > # Rewrite the hostlist to accept "," as a delimeter for hostnames >> too. >> > > > hostlist=`echo $hostlist | tr ',' ' '` >> > > > >> > > > is_host_up() { >> > > > for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >> > > > do >> > > > if >> > > > ping -w1 -c1 "$1" >/dev/null 2>&1 >> > > > then >> > > > sleep 10 >> > > > else >> > > > return 1 >> > > > fi >> > > > done >> > > > return 0 >> > > > } >> > > >> > > This is from external/ssh and used only if the parameter >> > > livedangerously is set to yes. Doesn't strike me as a very robust >> > > way to check if a node is running. Is there another way? Using >> > > the cblades sounds like a most natural way. >> > >> > >> > Yes. You're clearly right. By way of excuse I'd claim I was rather >> > narrowly focused on getting a server fenced and therefore I ignored >> other >> > functions. If you'll tolerate a "version 2, (or, I suppose v0.02), I'll >> try >> > again. It'll take me a few hours to fix and test, I'll try again. (A >> quick >> > experiment reveals that yes, the master blade does indeed know the >> status of >> > a given processor blade.) >> >> Great. >> >> > > > MASTER_BLADE=`$SSH_COMMAND cblade-1 "bin/get_master.sh" | tail -1` >> > > > >> > > > case $1 in >> > > > gethosts) >> > > > for h in $hostlist ; do >> > > > echo $h >> > > > done >> > > > exit 0 >> > > > ;; >> > > > on) >> > > > for h in $hostlist >> > > > do >> > > > if >> > > > [ "$h" != "$2" ] >> > > > then >> > > > continue >> > > > else >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE >> > > "bin/get_processor-blade.sh $h"` >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/on" >> > > > fi >> > > > sleep 30 >> > > > if >> > > > is_host_up $h >> > > > then >> > > > exit 1 >> > > > else >> > > > exit 0 >> > > > fi >> > > > done >> > > > exit 0 >> > > > ;; >> > > > off) >> > > > for h in $hostlist >> > > > do >> > > > if >> > > > [ "$h" != "$2" ] >> > > > then >> > > > continue >> > > > else >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE >> > > "bin/get_processor-blade.sh $h"` >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/off" >> > > > fi >> > > > done >> > > > exit 0 >> > > > ;; >> > > > reset) >> > > > for h in $hostlist >> > > > do >> > > > if >> > > > [ "$h" != "$2" ] >> > > > then >> > > > continue >> > > > else >> > > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE >> > > "bin/get_processor-blade.sh $h"` >> > > > $SSH_COMMAND $MASTER_BLADE "echo 'please' > >> > > /proc/egenera/ipmi/node/$P_BLADE/control/nmi" >> > > > fi >> > > > sleep 20 >> > > > if >> > > > is_host_up $h >> > > > then >> > > > exit 1 >> > > > else >> > > > exit 0 >> > > > fi >> > > > done >> > > > exit 1 >> > > > ;; >> > > >> > > on), off), and reset) cases are very similar, they should be made >> > > to share some code. Also, the sleep values look too ephemeral for >> > > my taste. There should be a relatively tight loop to check the >> > > host status. >> > > >> > >> > I'm not sure yet what to do about the "ephemeral" sleep values. I chose >> > arbitrarily long waits because egenera blades take a surprisingly long >> time >> > to boot. Your point is well taken, however, the code can be improved. >> >> Looking forward to it :) >> >> > > > status) >> > > > if >> > > > [ -z "$hostlist" ] >> > > > then >> > > > exit 1 >> > > > fi >> > > > for h in $hostlist >> > > > do >> > > > if >> > > > ping -w1 -c1 "$h" 2>&1 | grep "unknown host" >> > > >> > > The status should return the status of the device, not the status >> > > of a node. >> > >> > >> > ?? I must have misunderstood the existing code. Hmmmmm. The status of >> > this device is always "0" if there's a hostlist and if $MASTER_BLADE is >> set, >> > I think. >> >> And if the c-blade is reachable. Something like >> "ssh $MASTER_BLADE true" should do. >> >> > Thank you for taking the itme to look this over. I'll clean it up some >> and >> > try again. >> >> Many thanks. >> >> Cheers, >> >> Dejan >> >> > Be well, >> > >> > Karl >> > >> > > >> > > >> > > Cheers, >> > > >> > > Dejan >> > > >> > > I've cleaned this up some, and reworked much of it. It has been tested in production for a couple of weeks now, and it works properly as near as I can tell. (It has properly restarted a misbehaving server.) I could not see a way to get the existing IPMI script to work with Egenera, and I have abandoned using anything other than native Egenera commands to control the blades. These scripts require that ssh keys are set up for user root between the cluster servers and the c-blades. Code: #The two bash scripts here at the top go on each of the Egenera C-Blades. #the STONITH module goes in /usr/lib/stonith/plugins/external/egenera on #each of the servers in the cluster. ############################### cut here ###################################### #!/bin/bash # get_status.sh - returns the status of a pServer, given the LPAN and pServer #called as: # get_status.sh pServer LPAN_Name UP # or # get_status.sh pServer LPAN_Name DOWN #depending on what you want. This only works of the pServer is running the PAN Agent #because until the agent comes up, the server does not count as "Up" if [ ${3} = "UP" ] then CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} | grep "Booted" | grep "Up"` if [ -z "${CONDITION}" ] then echo "Down" # No Joy else echo "Up" # Joy! fi else # it should be the case that ${3} = "DOWN" CONDITION=`/opt/panmgr/bin/esh pserver -l ${2}/${1} | grep ^${1} | grep "Shutdown"` if [ -z "${CONDITION}" ] then echo "Up" # No Joy else echo "Down" # Joy! fi fi ############################### cut here ###################################### #!/bin/bash # get_master.sh - returns the name of the master control blade in the PAN. MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2` echo "cblade-${MASTER}" ############################### cut here ###################################### #!/bin/bash # # External STONITH module for Egenera pServers. # # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <[email protected]> # Copyright (c) 2009 Karl W. Lewis <karl.w.lewis - AT - gmail.com> # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # Further, this software is distributed without any warranty that it is # free of the rightful claim of any third person regarding infringement # or the like. Any license provided herein, whether implied or # otherwise, applies only to this software file. Patent licenses, if # any, provided herein do not apply to combinations of this program with # other software, or any other product whatsoever. # # You should have received a copy of the GNU General Public License # along with this program; if not, write the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. # SSH_COMMAND="/usr/bin/ssh -q -x -n -l root" GET_MASTER="/root/bin/get_master.sh" GET_STATUS="/root/bin/get_status.sh" PSERVER_COMMAND="/opt/panmgr/bin/esh pserver" #These TIMEOUT values were obtained from the 'pserver -l pan/pserver' command # on a given Egenera frame under "Timeouts" YMMV. BOOT_TIMEOUT=270 REBOOT_TIMEOUT=450 SHUTDOWN_TIMEOUT=180 TIMEOUT=${REBOOT_TIMEOUT} STATUS="UNKNOWN" #LOG=/var/log/egenera-stonith.log #echo `date` >> ${LOG} #echo ${hostlist} >> ${LOG} #echo ${lpan_name} >> ${LOG} #echo $1 >> ${LOG} #echo $2 >> ${LOG} #echo "executing--------------" >> ${LOG} # Rewrite the hostlist to accept "," as a delimeter for hostnames too. hostlist=`echo ${hostlist} | tr ',' ' '` is_host_up() { TIME=$((${TIMEOUT}/5)) for j in 1 2 3 4 5 6 do STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h} ${lpan_name} UP"` if [ "${STATUS}" = "Down" ] then sleep ${TIME} else SYSTEM_STATUS="On Target" return 1 fi done return 0 } is_host_down() { TIME=$((${TIMEOUT}/5)) for j in 1 2 3 4 5 6 do STATUS=`${SSH_COMMAND} ${MASTER_BLADE} "${GET_STATUS} ${h} ${lpan_name} DOWN"` if [ "${STATUS}" = "Up" ] then sleep ${TIME} else SYSTEM_STATUS="On Target" return 1 fi done return 0 } MASTER_BLADE=`${SSH_COMMAND} cblade-1 "${GET_MASTER}" | tail -1` if [ -z "${MASTER_BLADE}" ] then MASTER_BLADE=`${SSH_COMMAND} cblade-2 "${GET_MASTER}" | tail -1` fi case ${1} in gethosts) for h in ${hostlist} do echo ${h} done exit 0 ;; on|off|reboot) for h in ${hostlist} do if [ "${h}" != "${2}" ] then continue else if [ "${1}" = "on" ] then CMD="--boot" elif [ "${1}" = "off" ] then CMD="--shutdown --force" else #must be "reboot" CMD="--reboot --force" fi ${SSH_COMMAND} ${MASTER_BLADE} "${PSERVER_COMMAND} ${CMD} ${lpan_name}/${h}" if [ "${1}" = "on" ] then TIMEOUT=${BOOT_TIMEOUT} elif [ "${1}" = "reboot" ] then TIMEOUT=${REBOOT_TIMEOUT} else TIMEOUT=${SHUTDOWN_TIMEOUT} fi if [ "${1}" = "off" ] then is_host_down ${h} else is_host_up ${h} fi fi if [ "${SYSTEM_STATUS}" = "On Target" ] then exit 0 else exit 1 fi done exit 1 # We got here only if the host was not in the list.... ;; status) if [ -z "${hostlist}" ] || [ -z "${lpan_name}" ] || [ -z "${MASTER_BLADE}" ] then exit 1 fi exit 0 ;; getconfignames) echo "hostlist lpan_name" exit 0 ;; getinfo-devid) echo "Egenera STONITH device" exit 0 ;; getinfo-devname) echo "Egenera STONITH external device" exit 0 ;; getinfo-devdescr) echo "Egenera-based Linux host reset" exit 0 ;; getinfo-devurl) echo "http://www.egenera.com"; exit 0 ;; getinfo-xml) cat << EGENERAXML <parameters> <parameter name="hostlist" unique="1" required="1"> <content type="string" /> <shortdesc lang="en"> Hostlist </shortdesc> <longdesc lang="en"> The list of hosts that the STONITH device controls </longdesc> </parameter> <parameter name="lpan_name" unique="1" required="1"> <content type="string" /> <shortdesc lang="en"> LPAN_Name </shortdesc> <longdesc lang="en"> The name of the LPAN that the cluster blades, that this STONITH device controls, live in </longdesc> </parameter> </parameters> EGENERAXML exit 0 ;; *) exit 1 ;; esac _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
