Hi All;
Context: I'm trying to setup a two node HA nfs server in a lab
environment. I want the systems to be smart enough to fail over if the
primary loses its (non-heartbeat) connection.
On a related note, if I've configured auto_failback to off, how do I
manually fail control back to the primary?
Topology:
eth0 on nfs1 and nfs2 are 192.168.0.180 and .181 respectively.
eth1 on nfs1 and nfs2 are 10.0.0.1 and .2 respectively. eth1 are joined
on a private switch.
192.168.0.1, the "reachability" host used in ipfail, is the default gateway.
Heartbeat is over the private ethernet link on eth1.
The details:
debian etch v 4.04r1, upgraded a few days ago. heartbeat-2 v2.0.7-2,
drbd0.7-utils v0.7.21-4, drbd0.7.21-4-module-source.
ha.cf: (cksum reports them to be identical)
nfs2:/etc/ha.d# grep -v "^#" ha.cf |grep [a-z]
debugfile /var/log/ha-debug
logfile /var/log/ha-log
bcast eth1 # Linux
auto_failback on
node nfs1 nfs2
respawn hacluster /usr/lib/heartbeat/ipfail
ping 192.168.0.1
haresources: (cksum reports them to be identical)
nfs1 IPaddr::192.168.0.182/24/eth0 drbddisk::r0
Filesystem::/dev/drbd0::/drbdtest::ext3 nfs-kernel-server
Symptoms:
If I unplug eth0 on nfs1 (the primary) so the ipfail test can no longer
ping its target - but nfs2 can - then the system fails over to nfs2
properly. However, just after that happens I get a log message saying
that "nfs1 can take our foreign resources", and it fails back to nfs1,
attempts and fails to unmount the file system, and goes through more or
less continual reboots.
Is this a bug in ipfail? Have I simply got things configured wrong? Do
I need a more recent copy of heartbeat?
Thanks,
--- David
nfs2:/var/log# cat ha-log
heartbeat[2308]: 2008/09/30_14:26:30 info: nfs1 wants to go standby [all]
heartbeat[2308]: 2008/09/30_14:28:26 WARN: node nfs1: is dead
heartbeat[2308]: 2008/09/30_14:28:26 info: Cancelling pending standby
operation
heartbeat[2308]: 2008/09/30_14:28:26 info: Dead node nfs1 gave up resources.
heartbeat[2308]: 2008/09/30_14:28:26 info: Resources being acquired from
nfs1.
heartbeat[2308]: 2008/09/30_14:28:26 info: Link nfs1:eth1 dead.
harc[2473]: 2008/09/30_14:28:26 info: Running /etc/ha.d/rc.d/status
status
heartbeat[2474]: 2008/09/30_14:28:26 info: No local resources
[/usr/lib/heartbeat/ResourceManager listkeys nfs2] to acquire.
mach_down[2492]: 2008/09/30_14:28:26 info: Taking over resource
group IPaddr::192.168.0.182/24/eth0
ResourceManager[2511]: 2008/09/30_14:28:26 info: Acquiring resource
group: nfs1 IPaddr::192.168.0.182/24/eth0 drbddisk::r0
Filesystem::/dev/drbd0::/drbdtest::ext3 nfs-kernel-server
IPaddr[2534]: 2008/09/30_14:28:26 INFO: IPaddr Resource is stopped
ResourceManager[2511]: 2008/09/30_14:28:26 info: Running
/etc/ha.d/resource.d/IPaddr 192.168.0.182/24/eth0 start
IPaddr[2730]: 2008/09/30_14:28:27 INFO: eval /sbin/ifconfig eth0:0
192.168.0.182 netmask 255.255.255.0 broadcast 192.168.0.255
IPaddr[2730]: 2008/09/30_14:28:27 INFO: Sending Gratuitous Arp for
192.168.0.182 on eth0:0 [eth0]
IPaddr[2730]: 2008/09/30_14:28:27 INFO: /usr/lib/heartbeat/send_arp -i
500 -r 10 -p /var/run/heartbeat/rsctmp/send_arp/send_arp-192.168.0.182
eth0 192.168.0.182 auto 192.168.0.182 ffffffffffff
IPaddr[2648]: 2008/09/30_14:28:27 INFO: IPaddr Success
ResourceManager[2511]: 2008/09/30_14:28:27 info: Running
/etc/ha.d/resource.d/drbddisk r0 start
Filesystem[2924]: 2008/09/30_14:28:27 INFO: Running status for
/dev/drbd0 on /drbdtest
Filesystem[2924]: 2008/09/30_14:28:27 INFO: /drbdtest is unmounted
(stopped)
Filesystem[2860]: 2008/09/30_14:28:27 INFO: Filesystem Resource is
stopped
ResourceManager[2511]: 2008/09/30_14:28:27 info: Running
/etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbdtest ext3 start
Filesystem[3028]: 2008/09/30_14:28:27 INFO: Running start for
/dev/drbd0 on /drbdtest
Filesystem[2964]: 2008/09/30_14:28:28 INFO: Filesystem Success
ResourceManager[2511]: 2008/09/30_14:28:28 info: Running
/etc/init.d/nfs-kernel-server start
mach_down[2492]: 2008/09/30_14:28:28 info:
/usr/lib/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[2492]: 2008/09/30_14:28:28 info: mach_down takeover
complete for node nfs1.
==> it looks like we've got a successful failover here:
heartbeat[2308]: 2008/09/30_14:28:28 info: mach_down takeover complete.
heartbeat[2308]: 2008/09/30_14:29:47 info: Heartbeat restart on node nfs1
heartbeat[2308]: 2008/09/30_14:29:47 info: Link nfs1:eth1 up.
heartbeat[2308]: 2008/09/30_14:29:47 info: Status update for node nfs1:
status init
heartbeat[2308]: 2008/09/30_14:29:47 info: Status update for node nfs1:
status up
harc[3130]: 2008/09/30_14:29:47 info: Running /etc/ha.d/rc.d/status
status
harc[3139]: 2008/09/30_14:29:47 info: Running /etc/ha.d/rc.d/status
status
heartbeat[2308]: 2008/09/30_14:30:18 info: Status update for node nfs1:
status active
harc[3148]: 2008/09/30_14:30:18 info: Running /etc/ha.d/rc.d/status
status
heartbeat[2308]: 2008/09/30_14:30:18 info: remote resource transition
completed.
heartbeat[2308]: 2008/09/30_14:30:18 info: nfs2 wants to go standby
[foreign]
==> but now nfs1 is running - despite ipfail not reaching its target -
and we want to fail back? This does not compute:(
heartbeat[2308]: 2008/09/30_14:30:19 info: standby: nfs1 can take our
foreign resources
heartbeat[3157]: 2008/09/30_14:30:19 info: give up foreign HA resources
(standby).
ResourceManager[3167]: 2008/09/30_14:30:19 info: Releasing resource
group: nfs1 IPaddr::192.168.0.182/24/eth0 drbddisk::r0
Filesystem::/dev/drbd0::/drbdtest::ext3 nfs-kernel-server
ResourceManager[3167]: 2008/09/30_14:30:19 info: Running
/etc/init.d/nfs-kernel-server stop
ResourceManager[3167]: 2008/09/30_14:30:19 info: Running
/etc/ha.d/resource.d/Filesystem /dev/drbd0 /drbdtest ext3 stop
Filesystem[3292]: 2008/09/30_14:30:19 INFO: Running stop for
/dev/drbd0 on /drbdtest
Filesystem[3292]: 2008/09/30_14:30:19 INFO: Trying to unmount
/drbdtest
Filesystem[3292]: 2008/09/30_14:30:19 ERROR: Couldn't unmount
/drbdtest; trying cleanup with SIGTERM
Filesystem[3292]: 2008/09/30_14:30:19 INFO: No processes on
/drbdtest were signalled
Filesystem[3292]: 2008/09/30_14:30:20 ERROR: Couldn't unmount
/drbdtest; trying cleanup with SIGTERM
Filesystem[3292]: 2008/09/30_14:30:20 INFO: No processes on
/drbdtest were signalled
Filesystem[3292]: 2008/09/30_14:30:21 ERROR: Couldn't unmount
/drbdtest; trying cleanup with SIGTERM
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
