Hi,
I have a two node cluster with pacemaker (heartbeat). Regularly after a cluster partition there is a problem with the membership. Both nodes see the other one offline and you can see the following log: Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: CRIT: Cluster node acsdupli-s returning after partition. Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: For information on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: WARN: Deadtime value may be too small. Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: See FAQ for information on tuning deadtime. Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: URL: http://linux-ha.org/wiki/FAQ#Heavy_Load Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: Link acsdupli-s:eth0 up. Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: WARN: Late heartbeat: Node acsdupli-s: interval 75040 ms Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: Status update for node acsdupli-s: status active Apr 20 14:13:18 ACSDUPLI-M cib: [13168]: WARN: cib_peer_callback: Discarding cib_apply_diff message (2ce2b) from acsdupli-s: not in our membership Apr 20 14:13:18 ACSDUPLI-M pingd: [13173]: notice: pingd_lstatus_callback: Status update: Ping node acsdupli-s now has status [up] Apr 20 14:13:18 ACSDUPLI-M pingd: [13173]: notice: pingd_nstatus_callback: Status update: Ping node acsdupli-s now has status [up] Apr 20 14:13:18 ACSDUPLI-M pingd: [13173]: notice: pingd_nstatus_callback: Status update: Ping node acsdupli-s now has status [active] Apr 20 14:13:18 ACSDUPLI-M crmd: [13172]: notice: crmd_ha_status_callback: Status update: Node acsdupli-s now has status [active] (DC=true) Apr 20 14:13:18 ACSDUPLI-M crmd: [13172]: info: crm_update_peer_proc: acsdupli-s.ais is now online Apr 20 14:13:19 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no mbr_track info Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: instance=91, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: no mbr_track info Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: Processing CCM event=NEW MEMBERSHIP (id=91) Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: instance=91, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=91) Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: NEW MEMBERSHIP: trans=91, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: CURRENT: acsdupli-m [nodeid=0, born=91] Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: populate_cib_nodes_ha: Requesting the list of configured nodes Apr 20 14:13:19 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no mbr_track info Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: instance=92, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: Processing CCM event=NEW MEMBERSHIP (id=92) Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1001, version=0.185.2): ok (rc=0) Apr 20 14:13:20 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no mbr_track info Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: instance=93, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: Processing CCM event=NEW MEMBERSHIP (id=93) Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: WARN: get_uuid: Could not calculate UUID for 10.102.47.65 Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: ERROR: crm_abort: add_node_copy: Triggered assert at xml.c:267 : src_node != NULL Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: no mbr_track info Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: instance=92, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=92) Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: NEW MEMBERSHIP: trans=92, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: CURRENT: acsdupli-m [nodeid=0, born=92] Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: populate_cib_nodes_ha: Requesting the list of configured nodes Apr 20 14:13:21 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no mbr_track info Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: instance=94, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: Processing CCM event=NEW MEMBERSHIP (id=94) Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: WARN: get_uuid: Could not calculate UUID for 10.102.47.65 Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: ERROR: crm_abort: add_node_copy: Triggered assert at xml.c:267 : src_node != NULL Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/1003, version=0.185.2): ok (rc=0) Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: no mbr_track info Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: instance=93, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 Ect ….... We must power off both servers and then power on the servers to clear the problem. Thank you for your help. Sincerely, Jean-Baptiste PS: See configuration below: Redhat Enterprise Linux Server 5.6 Pacemaker 1.0.10 Heartbeat 3.0.4 Cluster-glue 1.0.7 CIB : <crm_config> <cluster_property_set id="cib-bootstrap-options"> <nvpair id="cib-bootstrap-options-default-action-timeout" name="default-action-timeout" value="300s"/> <nvpair id="cib-bootstrap-options-default-resource-stickiness" name="default-resource-stickiness" value="100"/> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/> <nvpair id="cib-bootstrap-options-dc-deadtime" name="dc-deadtime" value="10s"/> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3"/> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="Heartbeat"/> </cluster_property_set> </crm_config> <constraints> <rsc_location id="location_pingd" rsc="mygroup "> <rule id="prefered_location_pingd" score="-INFINITY"> <expression attribute="pind" id="prefered_location_pingd-pind" operation="not_defined"/> <expression attribute="pingd" id="prefered_location_pingd-pingd" operation="lte" value="0"/> </rule> </rsc_location> </constraints> HA.CF logfile /var/log/ha-log logfacility local0 keepalive 1 deadtime 10 warntime 5 initdead 30 udpport 694 apiauth mgmtd uid=root respawn root /usr/lib/heartbeat/mgmtd -v crm yes ucast eth0 10.102.47.85 ucast eth0 10.102.47.86 respawn root /usr/lib/heartbeat/pingd -m 4000 -d 20s -a pingd ping 10.102.47.65 node ACSDUPLI-M node ACSDUPLI-S auto_failback OFF _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
