Gentlemen, I have troubles with NFS failover times when the NFS clients are not located in the same network as the HA NFS server (the connection between the nfs client and the nfs server is routed and not switched).
The HA NFS server is a Pacemaker+Corosync cluster which consists of 2 NFS nodes (active/passive using ocf::heartbeat:exportfs) plus one quorum node. All nodes are in the same network / VLAN. Here is my Pacemaker configuration: http://pastebin.com/WiAmQQwX My NFS clients are in several different networks / VLANs. All these VLANs are managed by / connected to one single L3 switch - Cisco Catalyst 6509. I'm testing failover with two NFS clients: - client-A with address in the same VLAN as the HA NFS server address. - client-B with address in a different VLAN than as the HA NFS server address. When I "crm resource migrate" my nfs group from the active cluster node (irvine) to the second node (mitchell) (the group consists of ocf:heartbeat:LVMm, ocf:heartbeat:Filesystem, ocf:heartbeat:exportfs and ocf:heartbeat:IPaddr2), I see: the I/O operation on client-A hangs for 10 - 15 seconds, then keeps working. Client A is in the same VLAN as the NFS server. the I/O operation on client-B hangs for about 5 minutes (sic!), then keeps working. Client B is in a different VLAN, it's gets routed (within the same switch) to reach the NFS server. These times above happen with the clients using NFSv3 mount. If I mount NFSv4, client B hangs for 60 seconds (as opposed to 5 minutes when using NFSv3). Yes, you would say this must be the ARP cache. What is the MYSTERY HERE IS: within these 5 minutes when client-B is hanging, starting at the moment I moved the nfs group from node irvine to node mitchell: - when I jump on client-B (10.0.12.132) and run "ssh 10.0.10.103" (the floating NFS server address managed by the cluster), I reach the right node (mitchell) - when I run "tcpdump -vv client-B" on irvine, which does not have the "10.0.10.103" floating address anymore (it's got moved to mitchel), I can see: [root@irvine ~]# tcpdump -nvv host 10.0.10.103 16:52:09.487087 IP (tos 0x0, ttl 63, id 34159, offset 0, flags [DF], proto TCP (6), length 176) 10.0.12.132.2324775663 > 10.0.10.103.2049: 120 getattr fh Unknown/01000101650000006D0F0A000C38CF534F1F2A5F000000004FC6F267337BD97D That tells me the L3 switch (cisco 6509) somehow remembers that there is an established connection between its two ports (to explain the tcpdump), but when a new connection is made (my ssh 10.0.10.103), it does a proper I'm jumping on the switch and running: #show ip arp 10.0.10.103 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.0.10.103 1 b499.bab7.01ca ARPA Vlan10 which is the MAC address of mitchell (good), but I can still see some tcpdump packets of the "10.0.10.103" on irvine (above). Anybody can send me the right direction, please, how to solve the 5 minutes NFSv3 hang window mystery? I suspect the reason NFSv4 clients hang for 60 seconds is that NFSv4 uses only one port and TCP, while NFSv3 uses both UDP and TCP and many ports? But still, 60 seconds is still too long and I need NFSv3 too. Thank you very much. Marji -- Marji Cermak, Drupal Systems Engineer, cermakm [at] gmail, www.marginaldrop.com _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
