Hi,
On Wed, Aug 26, 2009 at 01:11:24PM -0400, Karl W. Lewis wrote:
> On Wed, Aug 26, 2009 at 10:58 AM, Dejan Muhamedagic
> <[email protected]>wrote:
>
> > Hi,
> >
> > On Mon, Aug 24, 2009 at 11:35:15AM -0400, Karl W. Lewis wrote:
> > > I don't know if anyone else is running Heartbeat/Pacemaker on an Egenera
> > > BladeFrame, but if they are, these scripts might be useful to help set up
> > > stonith.
> > >
> > > The Egeneras use an IPMI arrangement to control the various processor
> > > blades, and these routines take advantage of that.
> >
> > Did you try ipmilan or external/ipmi too?
>
>
> I did not, but a quick review leads me to believe that the egenera frame
> does not count as an impi "device". One might be able to make that work,
> but you'd have to modify the ipmi script until it looked much like this
> example, or, you'd have to play very. very hard with the egnera c-blades,
> and I can't imagine anyone would want to do that.
Well, either they are IPMI compliant or not, i.e. our ipmi
stonith modules should work without modifications. I think it's
worth trying. At any rate, I think that this is a different
solution worth pursuing, since it would reduce the configuration
quite a bit.
> > > This may not be the best method of doing it, but it does appear to work.
> > > It is not fast. What it lacks in speed, however, it makes up for in
> > > flexibility. It can figure out which c-blade is the master, and then
> > figure
> > > out which p-blade is running the server to be stonithed, and then
> > stonith,
> > > it. The framework for this was an existing external stonith script that
> > I
> > > simply modified to meet my needs.
> > >
> > > The two short shell scripts at the top need to live in /root/bin/ on both
> >
> > I guess you wanted to say $HOME/bin. Note that the root user may
> > not always have the same home directory. Perhaps it would be
> > better to go with an absolute path such as /usr/local/bin.
>
>
> I actually put them in root's directory with malice aforethought, but now
> that you've asked the question... I guess I can't defend that decision. I
> use the user root because that's the only user on the c-blades under
> egenera's setup. One could add another user, but I can't advocate changing
> the config of the c-blades.
>
>
> >
> >
> > > control blades. The cluster servers need to have ssh keys and any
> > required
> > > firewall rules setup such that root on the cluster servers can ssh to the
> > > c-blades.
> >
> > There can be more than one? Is that covered in any way by the
> > script, i.e. does it try all available?
>
>
> Yes. Just so. There are two c-baldes; one is master and one is standby. I
> *think* that the master is the only one that can issue the ipmi commands and
> get a reaction. I've not experimented with that, however. (I hit a
> solution that worked, and I stopped experimenting.)
It would be good if you do some :) You don't want any surprises
when it comes to fencing.
> The script tracks down
> which of the two is master and then talks to that server. One could avoid
> that by giving both c-blades the same set of rsa host keys, and then just
> talking to the virtual IP Address that the master blade, which ever of the
> two it is, is answering on. Again, I am disinclined to modify the egenera
> setup that way. I tried telling ssh to ignore the key verification, but it
> still updates the knownhosts file when the script logs in, so it'll throw
> errors the next time it tries if the master has moved.
Right. Isn't there an option to tell ssh not to update the
knownhosts file.
> > > If anyone else is already doing this and has a better way, I'd be happy
> > to
> > > hear about it.
> > >
> > > Karl
> >
> > > ### THESE TWO SHELL SCRIPTS ARE EXPECTED IN /root/bin ON BOTH c-blades
> > > ### THAT WAY THE SCRPIT THAT FOLLOWS WILL WORK REGARDLESS OF WHICH
> > > ### EGENERA c-blade IS PAN-MASTER
> > >
> > > #!/bin/bash
> > > # get_master.sh - returns the name of the master control blade in the
> > > PAN.
> > >
> > > MASTER=`/opt/panmgr/bin/esh pan | grep ^c | grep Master | cut -c 2`
> > > echo "cblade-${MASTER}"
> > > #CUT HERE #############################################################
> > >
> > > #!/bin/bash
> > >
> > > # get_processor_blade.sh - identifies the blade running a given pserver.
> >
> > This name doesn't exactly match the references in the script.
>
>
>
> :%s/_blade/-blade/
>
> I'm a lousy typist....
>
>
>
> >
> >
> > > BLADE=`/opt/panmgr/bin/esh pan | grep ^${1}`
> > > echo ${BLADE} | sed -e "s/^.*MyBladeFrame\/\(.\)\(.\) .*/a\1-\2/"
> > > #CUT HERE #############################################################
> > >
> > > #!/bin/bash
> > > #
> > > # External STONITH module for Egenera.
> > > #
> > > # Copyright (c) 2004 SUSE LINUX AG - Lars Marowsky-Bree <[email protected]>
> > > # Copyright (c) 2009 Karl W. Lewis <[email protected]>
> > > # This program is free software; you can redistribute
> > > #it and/or modify # it under the terms of version 2 of the GNU General
> > > #Public License as # published by the Free Software Foundation.
> > > #
> > > # This program is distributed in the hope that it would be useful, but
> > > # WITHOUT ANY WARRANTY; without even the implied warranty of
> > > # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> > > #
> > > # Further, this software is distributed without any warranty that it is
> > > # free of the rightful claim of any third person regarding infringement
> > > # or the like. Any license provided herein, whether implied or
> > > # otherwise, applies only to this software file. Patent licenses, if
> > > # any, provided herein do not apply to combinations of this program with
> > > # other software, or any other product whatsoever.
> > > #
> > > # You should have received a copy of the GNU General Public License
> > > # along with this program; if not, write the Free Software Foundation,
> > > # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
> > > #
> > >
> > > SSH_COMMAND="/usr/bin/ssh"
> > >
> > > #LOG=/var/log/egenera-stonith.log
> > > #echo `date` >> $LOG
> > > #echo $hostlist >> $LOG
> > > #echo $1 >> $LOG
> > > #echo $2 >> $LOG
> > > #echo "executing--------------" >> $LOG
> > >
> > > # Rewrite the hostlist to accept "," as a delimeter for hostnames too.
> > > hostlist=`echo $hostlist | tr ',' ' '`
> > >
> > > is_host_up() {
> > > for j in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> > > do
> > > if
> > > ping -w1 -c1 "$1" >/dev/null 2>&1
> > > then
> > > sleep 10
> > > else
> > > return 1
> > > fi
> > > done
> > > return 0
> > > }
> >
> > This is from external/ssh and used only if the parameter
> > livedangerously is set to yes. Doesn't strike me as a very robust
> > way to check if a node is running. Is there another way? Using
> > the cblades sounds like a most natural way.
>
>
> Yes. You're clearly right. By way of excuse I'd claim I was rather
> narrowly focused on getting a server fenced and therefore I ignored other
> functions. If you'll tolerate a "version 2, (or, I suppose v0.02), I'll try
> again. It'll take me a few hours to fix and test, I'll try again. (A quick
> experiment reveals that yes, the master blade does indeed know the status of
> a given processor blade.)
Great.
> > > MASTER_BLADE=`$SSH_COMMAND cblade-1 "bin/get_master.sh" | tail -1`
> > >
> > > case $1 in
> > > gethosts)
> > > for h in $hostlist ; do
> > > echo $h
> > > done
> > > exit 0
> > > ;;
> > > on)
> > > for h in $hostlist
> > > do
> > > if
> > > [ "$h" != "$2" ]
> > > then
> > > continue
> > > else
> > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE
> > "bin/get_processor-blade.sh $h"`
> > > $SSH_COMMAND $MASTER_BLADE "echo 'please' >
> > /proc/egenera/ipmi/node/$P_BLADE/control/on"
> > > fi
> > > sleep 30
> > > if
> > > is_host_up $h
> > > then
> > > exit 1
> > > else
> > > exit 0
> > > fi
> > > done
> > > exit 0
> > > ;;
> > > off)
> > > for h in $hostlist
> > > do
> > > if
> > > [ "$h" != "$2" ]
> > > then
> > > continue
> > > else
> > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE
> > "bin/get_processor-blade.sh $h"`
> > > $SSH_COMMAND $MASTER_BLADE "echo 'please' >
> > /proc/egenera/ipmi/node/$P_BLADE/control/off"
> > > fi
> > > done
> > > exit 0
> > > ;;
> > > reset)
> > > for h in $hostlist
> > > do
> > > if
> > > [ "$h" != "$2" ]
> > > then
> > > continue
> > > else
> > > P_BLADE=`$SSH_COMMAND $MASTER_BLADE
> > "bin/get_processor-blade.sh $h"`
> > > $SSH_COMMAND $MASTER_BLADE "echo 'please' >
> > /proc/egenera/ipmi/node/$P_BLADE/control/nmi"
> > > fi
> > > sleep 20
> > > if
> > > is_host_up $h
> > > then
> > > exit 1
> > > else
> > > exit 0
> > > fi
> > > done
> > > exit 1
> > > ;;
> >
> > on), off), and reset) cases are very similar, they should be made
> > to share some code. Also, the sleep values look too ephemeral for
> > my taste. There should be a relatively tight loop to check the
> > host status.
> >
>
> I'm not sure yet what to do about the "ephemeral" sleep values. I chose
> arbitrarily long waits because egenera blades take a surprisingly long time
> to boot. Your point is well taken, however, the code can be improved.
Looking forward to it :)
> > > status)
> > > if
> > > [ -z "$hostlist" ]
> > > then
> > > exit 1
> > > fi
> > > for h in $hostlist
> > > do
> > > if
> > > ping -w1 -c1 "$h" 2>&1 | grep "unknown host"
> >
> > The status should return the status of the device, not the status
> > of a node.
>
>
> ?? I must have misunderstood the existing code. Hmmmmm. The status of
> this device is always "0" if there's a hostlist and if $MASTER_BLADE is set,
> I think.
And if the c-blade is reachable. Something like
"ssh $MASTER_BLADE true" should do.
> Thank you for taking the itme to look this over. I'll clean it up some and
> try again.
Many thanks.
Cheers,
Dejan
> Be well,
>
> Karl
>
> >
> >
> > Cheers,
> >
> > Dejan
> >
> > > then
> > > exit 1
> > > fi
> > > done
> > > exit 0
> > > ;;
> > > getconfignames)
> > > echo "hostlist"
> > > exit 0
> > > ;;
> > > getinfo-devid)
> > > echo "Egenera STONITH device"
> > > exit 0
> > > ;;
> > > getinfo-devname)
> > > echo "Egenera STONITH external device"
> > > exit 0
> > > ;;
> > > getinfo-devdescr)
> > > echo "Egenera-based Linux host reset"
> > > exit 0
> > > ;;
> > > getinfo-devurl)
> > > echo "http://www.egenera.com";
> > > exit 0
> > > ;;
> > > getinfo-xml)
> > > cat << EGENERAXML
> > > <parameters>
> > > parameter name="hostlist" unique="1" required="1"> <contenttype="string"
> > />
> > > <shortdesc lang="en"> Hostlist </shortdesc>
> > > <longdesc lang="en"> The list of hosts that the STONITH device controls
> > </longdesc>
> > > </parameter>
> > > </parameters>
> > > EGENERAXML
> > > exit 0
> > > ;;
> > > *)
> > > exit 1
> > > ;;
> > > esac
> > >
> > >
> > >
> > >
> >
> > > _______________________________________________
> > > Linux-HA mailing list
> > > [email protected]
> > > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > > See also: http://linux-ha.org/ReportingProblems
> > _______________________________________________
> > Linux-HA mailing list
> > [email protected]
> > http://lists.linux-ha.org/mailman/listinfo/linux-ha
> > See also: http://linux-ha.org/ReportingProblems
> >
> _______________________________________________
> Linux-HA mailing list
> [email protected]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
