Just a quick follow up - I had this answered on the Corosync mailing list (which I guess should have been the place for this anyway). As I was blocking all traffic with iptables, it was also blocking lo, which caused all sorts of things to break. As soon as I only blocked on eth0, things started working as expected.
-----Original Message----- From: Mark Round [mailto:mark.ro...@nccgroup.com] Sent: 05 September 2013 11:44 To: pacemaker@oss.clusterlabs.org Subject: [Pacemaker] Corosync quorum not updating on split node Hi all, I have a problem whereby when I create a network split/partition (by dropping traffic with iptables), the victim node for some reason does not realise it has split from the network. It seems to recognise that it can't form a cluster due to network issues, but the status is not reflected in the output from corosync-quorumtool, and cluster services (via pacemaker) still continue to run. However, the other nodes in the rest of the cluster do realise they have lost contact with a node, no longer have quorum and correctly shut down services. When I block traffic on the victim node's eth0, The remaining nodes see that they cannot communicate with it and shutdown : # corosync-quorumtool -s Version: 1.4.5 Nodes: 3 Ring ID: 696 Quorum type: corosync_votequorum Quorate: No Node votes: 1 Expected votes: 7 Highest expected: 7 Total votes: 3 Quorum: 4 Activity blocked Flags: However, the victim node still thinks everything is fine, and maintains a view of the cluster prior to the split : # corosync-quorumtool -s Version: 1.4.5 Nodes: 4 Ring ID: 716 Quorum type: corosync_votequorum Quorate: Yes Node votes: 1 Expected votes: 7 Highest expected: 7 Total votes: 4 Quorum: 4 Flags: Quorate However, it does notice in the logs that it cannot now form cluster, as the following messages repeat constantly : corosync [MAIN ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly. I would expect at this point for it to be in it's own network partition with a total of 1 vote, and block activity. However, this does not seem to happen until just after it rejoins the cluster. When I unblock traffic and it rejoins, I see the victim finally realise it had lost quorum : Sep 05 09:52:21 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 720: memb=1, new=0, lost=3 Sep 05 09:52:21 corosync [VOTEQ ] quorum lost, blocking activity Sep 05 09:52:21 corosync [QUORUM] This node is within the non-primary component and will NOT provide any services. Sep 05 09:52:21 corosync [QUORUM] Members[1]: 358898186 And a second or so later it regains quorum : crmd: notice: ais_dispatch_message: Membership 736: quorum acquired So my question is why, when it realises it cannot form a cluster ("Totem in unable to form..."), does it not loose quorum, update the status as reflected by quorumtool and shutdown cluster services ? Configuration file example and package versions/environment listed below. I'm using "updu" protocol as we need to avoid multicast in this environment; it will eventually be using a routed network. This behaviour also persists when I disable the pacemaker plugin and just test with corosync. compatibility: whitetank totem { version: 2 secauth: off interface { member { memberaddr: 10.90.100.20 } member { memberaddr: 10.90.100.21 } ... ... more nodes snipped ... ringnumber: 0 bindnetaddr: 10.90.100.20 mcastport: 5405 } transport: udpu } amf { mode: disabled } aisexec { user: root group: root } quorum { provider: corosync_votequorum expected_votes: 7 } service { # Load the Pacemaker Cluster Resource Manager name: pacemaker ver: 0 } Environment : CentOS 6.4 Packages from OpenSUSE : http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/RedHat_RHEL-6/x86_64/ # rpm -qa | egrep "^(cluster|corosync|crm|libqb|pacemaker|resource-agents)" | sort cluster-glue-1.0.11-3.1.x86_64 cluster-glue-libs-1.0.11-3.1.x86_64 corosync-1.4.5-2.2.x86_64 corosynclib-1.4.5-2.2.x86_64 crmsh-1.2.6-0.rc3.3.1.x86_64 libqb0-0.14.4-1.2.x86_64 pacemaker-1.1.9-2.1.x86_64 pacemaker-cli-1.1.9-2.1.x86_64 pacemaker-cluster-libs-1.1.9-2.1.x86_64 pacemaker-libs-1.1.9-2.1.x86_64 resource-agents-3.9.5-3.1.x86_64 Regards, -Mark ________________________________ Mark Round Senior Systems Administrator NCC Group Kings Court Kingston Road Leatherhead, KT22 7SL Telephone: +44 1372 383815 Mobile: +44 7790 770413 Fax: Website: www.nccgroup.com<http://www.nccgroup.com> Email: mark.ro...@nccgroup.com<mailto:mark.ro...@nccgroup.com> [http://www.nccgroup.com/media/192418/nccgrouplogo.jpg] <http://www.nccgroup.com/> ________________________________ This email is sent for and on behalf of NCC Group. NCC Group is the trading name of NCC Group Performance Testing Limited (Registered in England CRN: 4069379). Registered Office: Manchester Technology Centre, Oxford Road, Manchester, M1 7EF. The ultimate holding company is NCC Group plc (Registered in England CRN: 4627044). Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than NCC Group or the sender of its existence. For more information about NCC Group please visit www.nccgroup.com<http://www.nccgroup.com> P Before you print think about the ENVIRONMENT For more information please visit <a href="http://www.mimecast.com";>http://www.mimecast.com<br> This email message has been delivered safely and archived online by Mimecast. </a> _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ________________________________ Mark Round Senior Systems Administrator NCC Group Kings Court Kingston Road Leatherhead, KT22 7SL Telephone: +44 1372 383815 Mobile: +44 7790 770413 Fax: Website: www.nccgroup.com<http://www.nccgroup.com> Email: mark.ro...@nccgroup.com<mailto:mark.ro...@nccgroup.com> [http://www.nccgroup.com/media/192418/nccgrouplogo.jpg] <http://www.nccgroup.com/> ________________________________ This email is sent for and on behalf of NCC Group. NCC Group is the trading name of NCC Group Performance Testing Limited (Registered in England CRN: 4069379). Registered Office: Manchester Technology Centre, Oxford Road, Manchester, M1 7EF. The ultimate holding company is NCC Group plc (Registered in England CRN: 4627044). Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than NCC Group or the sender of its existence. For more information about NCC Group please visit www.nccgroup.com<http://www.nccgroup.com> P Before you print think about the ENVIRONMENT For more information please visit <a href="http://www.mimecast.com";>http://www.mimecast.com<br> This email message has been delivered safely and archived online by Mimecast. </a> _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
