Hi, On Thu, Apr 21, 2011 at 04:52:23PM +0200, Jean-Baptiste GIRARD wrote: > Hi, > > > > I have a two node cluster with pacemaker (heartbeat). > > Regularly after a cluster partition there is a problem with the membership. > Both nodes see the other one offline and you can see the following log: > > > > > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: CRIT: Cluster node acsdupli-s > returning after partition. > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: For information on > cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: WARN: Deadtime value may be > too small. > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: See FAQ for information > on tuning deadtime. > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: URL: > http://linux-ha.org/wiki/FAQ#Heavy_Load > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: Link acsdupli-s:eth0 > up. > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: WARN: Late heartbeat: Node > acsdupli-s: interval 75040 ms > > Apr 20 14:13:18 ACSDUPLI-M heartbeat: [13151]: info: Status update for node > acsdupli-s: status active > > Apr 20 14:13:18 ACSDUPLI-M cib: [13168]: WARN: cib_peer_callback: Discarding > cib_apply_diff message (2ce2b) from acsdupli-s: not in our membership > > Apr 20 14:13:18 ACSDUPLI-M pingd: [13173]: notice: pingd_lstatus_callback: > Status update: Ping node acsdupli-s now has status [up] > > Apr 20 14:13:18 ACSDUPLI-M pingd: [13173]: notice: pingd_nstatus_callback: > Status update: Ping node acsdupli-s now has status [up] > > Apr 20 14:13:18 ACSDUPLI-M pingd: [13173]: notice: pingd_nstatus_callback: > Status update: Ping node acsdupli-s now has status [active] > > Apr 20 14:13:18 ACSDUPLI-M crmd: [13172]: notice: crmd_ha_status_callback: > Status update: Node acsdupli-s now has status [active] (DC=true) > > Apr 20 14:13:18 ACSDUPLI-M crmd: [13172]: info: crm_update_peer_proc: > acsdupli-s.ais is now online > > Apr 20 14:13:19 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no > mbr_track info > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: > instance=91, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: no > mbr_track info > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: > Processing CCM event=NEW MEMBERSHIP (id=91) > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: > instance=91, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: crmd_ccm_msg_callback: > Quorum (re)attained after event=NEW MEMBERSHIP (id=91) > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: NEW > MEMBERSHIP: trans=91, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: > CURRENT: acsdupli-m [nodeid=0, born=91] > > Apr 20 14:13:19 ACSDUPLI-M crmd: [13172]: info: populate_cib_nodes_ha: > Requesting the list of configured nodes > > Apr 20 14:13:19 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no > mbr_track info > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: mem_handle_event: > instance=92, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:19 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: > Processing CCM event=NEW MEMBERSHIP (id=92) > > Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: cib_process_request: > Operation complete: op cib_modify for section nodes (origin=local/crmd/1001, > version=0.185.2): ok (rc=0) > > Apr 20 14:13:20 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster > > Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > > Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no > mbr_track info > > Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > > Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: mem_handle_event: > instance=93, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:20 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: > Processing CCM event=NEW MEMBERSHIP (id=93) > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: WARN: get_uuid: Could not > calculate UUID for 10.102.47.65
This looks odd. Do all nodes have uuid defined? Not sure how exactly crmd gets the node UUID. Normally, it's recorded in hostcache and hb_uuid.txt and in the CIB in the id attribute of the node element. The CIB below contains no nodes section and it's not compliant with the RNG schema. > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: ERROR: crm_abort: add_node_copy: > Triggered assert at xml.c:267 : src_node != NULL I guess that here crmd aborted, perhaps because previously it couldn't get the UUID. Thanks, Dejan > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: no > mbr_track info > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: > instance=92, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: crmd_ccm_msg_callback: > Quorum (re)attained after event=NEW MEMBERSHIP (id=92) > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: NEW > MEMBERSHIP: trans=92, nodes=1, new=0, lost=0 n_idx=0, new_idx=1, old_idx=3 > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: ccm_event_detail: > CURRENT: acsdupli-m [nodeid=0, born=92] > > Apr 20 14:13:20 ACSDUPLI-M crmd: [13172]: info: populate_cib_nodes_ha: > Requesting the list of configured nodes > > Apr 20 14:13:21 ACSDUPLI-M ccm: [13167]: info: Break tie for 2 nodes cluster > > Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: no > mbr_track info > Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: mem_handle_event: > instance=94, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: cib_ccm_msg_callback: > Processing CCM event=NEW MEMBERSHIP (id=94) > Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: WARN: get_uuid: Could not > calculate UUID for 10.102.47.65 > Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: ERROR: crm_abort: add_node_copy: > Triggered assert at xml.c:267 : src_node != NULL > Apr 20 14:13:21 ACSDUPLI-M cib: [13168]: info: cib_process_request: > Operation complete: op cib_modify for section nodes (origin=local/crmd/1003, > version=0.185.2): ok (rc=0) > Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an > event OC_EV_MS_INVALID from ccm > Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: no > mbr_track info > Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: Got an > event OC_EV_MS_NEW_MEMBERSHIP from ccm > Apr 20 14:13:21 ACSDUPLI-M crmd: [13172]: info: mem_handle_event: > instance=93, nodes=1, new=0, lost=0, n_idx=0, new_idx=1, old_idx=3 > > > Ect ….... > > > > > > > > We must power off both servers and then power on the servers to clear the > problem. > > > > Thank you for your help. > > > > Sincerely, > > Jean-Baptiste > > > > > > PS: See configuration below: > > > > Redhat Enterprise Linux Server 5.6 > > Pacemaker 1.0.10 > > Heartbeat 3.0.4 > > Cluster-glue 1.0.7 > > > > CIB : > > <crm_config> > > <cluster_property_set id="cib-bootstrap-options"> > > <nvpair id="cib-bootstrap-options-default-action-timeout" > name="default-action-timeout" value="300s"/> > > <nvpair id="cib-bootstrap-options-default-resource-stickiness" > name="default-resource-stickiness" value="100"/> > > <nvpair id="cib-bootstrap-options-stonith-enabled" > name="stonith-enabled" value="false"/> > > <nvpair id="cib-bootstrap-options-dc-deadtime" name="dc-deadtime" > value="10s"/> > > <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" > value="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3"/> > > <nvpair id="cib-bootstrap-options-cluster-infrastructure" > name="cluster-infrastructure" value="Heartbeat"/> > > </cluster_property_set> > > </crm_config> > > > > <constraints> > > <rsc_location id="location_pingd" rsc="mygroup "> > > <rule id="prefered_location_pingd" score="-INFINITY"> > > <expression attribute="pind" id="prefered_location_pingd-pind" > operation="not_defined"/> > > <expression attribute="pingd" id="prefered_location_pingd-pingd" > operation="lte" value="0"/> > > </rule> > > </rsc_location> > > </constraints> > > > > > > HA.CF > > logfile /var/log/ha-log > > logfacility local0 > > keepalive 1 > > deadtime 10 > > warntime 5 > > initdead 30 > > udpport 694 > > apiauth mgmtd uid=root > > respawn root /usr/lib/heartbeat/mgmtd -v > > crm yes > > ucast eth0 10.102.47.85 > > ucast eth0 10.102.47.86 > > respawn root /usr/lib/heartbeat/pingd -m 4000 -d 20s -a pingd > > ping 10.102.47.65 > > node ACSDUPLI-M > > node ACSDUPLI-S > > auto_failback OFF > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
