On 3/1/12 12:10 PM, William Seligman wrote: > On 3/1/12 6:34 AM, emmanuel segura wrote: >> try to change the fence daemon tag like this >> ==================================== >> <fence_daemon clean_start="1" post_join_delay="30" /> >> ==================================== >> change your cluster config version and after reboot the cluster > > This did not change the behavior of the cluster. In particular, I'm still > dealing with this: > >>>>> - If the system starts with cman running, and I start drbd, it's >>>>> likely that system who is _not_ Outdated will be fenced (rebooted).
This just happened again. Here's the log from the "bad" node, the one I stopped and then restarted. cman is running (not pacemaker). I start drbd: Mar 1 12:03:49 orestes-tb kernel: drbd: initialized. Version: 8.3.12 (api:88/proto:86-96) Mar 1 12:03:49 orestes-tb kernel: drbd: GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by [email protected], 2012-02-28 18:01:34 Mar 1 12:03:49 orestes-tb kernel: drbd: registered as block device major 147 Mar 1 12:03:49 orestes-tb kernel: drbd: minor_table @ 0xffff88041dbc4b80 Mar 1 12:03:49 orestes-tb kernel: block drbd0: Starting worker thread (from cqueue [2942]) Mar 1 12:03:49 orestes-tb kernel: block drbd0: disk( Diskless -> Attaching ) Mar 1 12:03:50 orestes-tb kernel: block drbd0: Found 57 transactions (57 active extents) in activity log. Mar 1 12:03:50 orestes-tb kernel: block drbd0: Method to ensure write ordering: barrier Mar 1 12:03:50 orestes-tb kernel: block drbd0: max BIO size = 130560 Mar 1 12:03:50 orestes-tb kernel: block drbd0: Adjusting my ra_pages to backing device's (32 -> 768) Mar 1 12:03:50 orestes-tb kernel: block drbd0: drbd_bm_resize called with capacity == 5611549368 Mar 1 12:03:50 orestes-tb kernel: block drbd0: resync bitmap: bits=701443671 words=10960058 pages=21407 Mar 1 12:03:50 orestes-tb kernel: block drbd0: size = 2676 GB (2805774684 KB) Mar 1 12:03:50 orestes-tb kernel: block drbd0: bitmap READ of 21407 pages took 625 jiffies Mar 1 12:03:50 orestes-tb kernel: block drbd0: recounting of set bits took additional 86 jiffies Mar 1 12:03:50 orestes-tb kernel: block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map. Mar 1 12:03:50 orestes-tb kernel: block drbd0: disk( Attaching -> Outdated ) Mar 1 12:03:50 orestes-tb kernel: block drbd0: attached to UUIDs 878999EFCFBE8E08:0000000000000000:494B48826E41A2C2:494A48826E41A2C3 Mar 1 12:03:50 orestes-tb kernel: block drbd0: conn( StandAlone -> Unconnected ) Mar 1 12:03:50 orestes-tb kernel: block drbd0: Starting receiver thread (from drbd0_worker [2951]) Mar 1 12:03:50 orestes-tb kernel: block drbd0: receiver (re)started Mar 1 12:03:50 orestes-tb kernel: block drbd0: conn( Unconnected -> WFConnection ) Mar 1 12:03:51 orestes-tb kernel: block drbd0: Handshake successful: Agreed network protocol version 96 Mar 1 12:03:51 orestes-tb kernel: block drbd0: conn( WFConnection -> WFReportParams ) Mar 1 12:03:51 orestes-tb kernel: block drbd0: Starting asender thread (from drbd0_receiver [2965]) Mar 1 12:03:51 orestes-tb kernel: block drbd0: data-integrity-alg: <not-used> Mar 1 12:03:51 orestes-tb kernel: block drbd0: drbd_sync_handshake: Mar 1 12:03:51 orestes-tb kernel: block drbd0: self 878999EFCFBE8E08:0000000000000000:494B48826E41A2C2:494A48826E41A2C3 bits:0 flags:0 Mar 1 12:03:51 orestes-tb kernel: block drbd0: peer D40A1613FAE8F5E9:878999EFCFBE8E09:878899EFCFBE8E09:494B48826E41A2C3 bits:0 flags:0 Mar 1 12:03:51 orestes-tb kernel: block drbd0: uuid_compare()=-1 by rule 50 Mar 1 12:03:51 orestes-tb kernel: block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) Mar 1 12:03:53 orestes-tb kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID ) Mar 1 12:04:01 orestes-tb corosync[2296]: [TOTEM ] A processor failed, forming new configuration. Mar 1 12:04:03 orestes-tb corosync[2296]: [QUORUM] Members[1]: 2 Mar 1 12:04:03 orestes-tb corosync[2296]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Mar 1 12:04:03 orestes-tb kernel: dlm: closing connection to node 1 Mar 1 12:04:03 orestes-tb corosync[2296]: [CPG ] chosen downlist: sender r(0) ip(129.236.252.14) r(1) ip(192.168.100.6) ; members(old:2 left:1) Mar 1 12:04:03 orestes-tb corosync[2296]: [MAIN ] Completed service synchronization, ready to provide service. Mar 1 12:04:03 orestes-tb fenced[2350]: fencing node hypatia-tb.nevis.columbia.edu As near as I can tell, the "bad" node sees that the "good" node is Primary and UpToDate, goes into WFSyncUUID... and then corosync/cman cheerfully fences the "good" node. >> Il giorno 01 marzo 2012 12:28, William Seligman <[email protected] >>> ha scritto: >> >>> On 3/1/12 4:15 AM, emmanuel segura wrote: >>> >>>> can you show me your /etc/cluster/cluster.conf? >>>> >>>> because i think your problem it's a fencing-loop >>>> >>> >>> Here it is: >>> >>> /etc/cluster/cluster.conf: >>> >>> <?xml version="1.0"?> >>> <cluster config_version="17" name="Nevis_HA"> >>> <logging debug="off"/> >>> <cman expected_votes="1" two_node="1" /> >>> <clusternodes> >>> <clusternode >>> name="hypatia-tb.nevis.**columbia.edu<http://hypatia-tb.nevis.columbia.edu>" >>> nodeid="1"> >>> <altname >>> name="hypatia-private.nevis.**columbia.edu<http://hypatia-private.nevis.columbia.edu>" >>> port="5405" >>> mcast="226.94.1.1"/> >>> <fence> >>> <method name="pcmk-redirect"> >>> <device name="pcmk" >>> port="hypatia-tb.nevis.**columbia.edu<http://hypatia-tb.nevis.columbia.edu> >>> "/> >>> </method> >>> </fence> >>> </clusternode> >>> <clusternode >>> name="orestes-tb.nevis.**columbia.edu<http://orestes-tb.nevis.columbia.edu>" >>> nodeid="2"> >>> <altname >>> name="orestes-private.nevis.**columbia.edu<http://orestes-private.nevis.columbia.edu>" >>> port="5405" >>> mcast="226.94.1.1"/> >>> <fence> >>> <method name="pcmk-redirect"> >>> <device name="pcmk" >>> port="orestes-tb.nevis.**columbia.edu<http://orestes-tb.nevis.columbia.edu> >>> "/> >>> </method> >>> </fence> >>> </clusternode> >>> </clusternodes> >>> <fencedevices> >>> <fencedevice name="pcmk" agent="fence_pcmk"/> >>> </fencedevices> >>> <fence_daemon post_join_delay="30" /> >>> <rm disabled="1" /> >>> </cluster> >>> >>> >>> >>> Il giorno 01 marzo 2012 01:03, William Seligman<seligman@nevis.** >>>> columbia.edu <[email protected]> >>>> >>>>> ha scritto: >>>>> >>>> >>>> On 2/28/12 7:26 PM, Lars Ellenberg wrote: >>>>> >>>>>> On Tue, Feb 28, 2012 at 03:51:29PM -0500, William Seligman wrote: >>>>>> >>>>>>> <off-topic> >>>>>>> Sigh. I wish that were the reason. >>>>>>> >>>>>>> The reason why I'm doing dual-primary is that I've a got a >>>>>>> single-primary two-node cluster in production that simply doesn't >>>>>>> work. One node runs resources; the other sits and twiddles its >>>>>>> fingers; fine. But when primary goes down, secondary has trouble >>>>>>> starting up all the resources; when we've actually had primary >>>>>>> failures (UPS goes haywire, hard drive failure) the secondary often >>>>>>> winds up in a state in which it runs none of the significant >>>>>>> resources. >>>>>>> >>>>>>> With the dual-primary setup I have now, both machines are running >>>>>>> the resources that typically cause problems in my single-primary >>>>>>> configuration. If one box goes down, the other doesn't have to >>>>>>> failover anything; it's already running them. (I needed IPaddr2 >>>>>>> cloning to work properly for this to work, which is why I started >>>>>>> that thread... and all the stupider of me for missing that crucial >>>>>>> page in Clusters From Scratch.) >>>>>>> >>>>>>> My only remaining problem with the configuration is restoring a >>>>>>> fenced node to the cluster. Hence my tests, and the reason why I >>>>>>> started this thread. >>>>>>> </off-topic> > >>>>>>> >>>>>> >>>>>> Uhm, I do think that is exactly on topic. >>>>>> >>>>>> Rather fix your resources to be able to successfully take over, >>>>>> than add even more complexity. >>>>>> >>>>>> What resources would that be, >>>>>> and why are they not taking over? >>>>>> >>>>> >>>>> I can't tell you in detail, because the major snafu happened on a >>>>> production system after a power outage a few months ago. My goal was to >>>>> get the thing stable as quickly as possible. In the end, that turned >>>>> out to be a non-HA configuration: One runs corosync+pacemaker+drbd, >>>>> while the other just runs drbd. It works, in the sense that the users >>>>> get their e-mail. If there's a power outage, I have to bring things up >>>>> manually. >>>>> >>>>> So my only reference is the test-bench dual-primary setup I've got >>>>> now, which is exhibiting the same kinds of problems even though the OS >>>>> versions, software versions, and layout are different. This suggests >>>>> that the problem lies in the way I'm setting up the configuration. >>>>> >>>>> The problems I have seem to be in the general category of "the 'good >>>>> guy' gets fenced when the 'bad guy' gets into trouble." Examples: >>>>> >>>>> - Assuming I start out with two crashed nodes. If I just start up DRBD >>>>> and nothing else, the partitions sync quickly with no problems. >>>>> >>>>> - If the system starts with cman running, and I start drbd, it's >>>>> likely that system who is _not_ Outdated will be fenced (rebooted). >>>>> Same thing if cman+pacemaker is running. >>>>> >>>>> - Cloned ocf:heartbeat:exportfs resources are giving me problems as >>>>> well (which is why I tried making changes to that resource script). >>>>> Assume I start with one node running cman+pacemaker, and the other >>>>> stopped. I turned on the stopped node. This will typically result in >>>>> the running node being fenced, because it has it times out when >>>>> stopping the exportfs resource. >>>>> >>>>> Falling back to DRBD 8.3.12 didn't change this behavior. >>>>> >>>>> My pacemaker configuration is long, so I'll excerpt what I think are >>>>> the relevant pieces in the hope that it will be enough for someone to >>>>> say "You fool! This is covered in Pacemaker Explained page 56!" When >>>>> bringing up a stopped node, in order to restart AdminClone pacemaker >>>>> wants to stop ExportsClone, then Gfs2Clone, then ClvmdClone. As I said, >>>>> it's the failure to stop ExportMail on the running node that causes it >>>>> to be fenced. >>>>> >>>>> primitive AdminDrbd ocf:linbit:drbd \ >>>>> params drbd_resource="admin" \ >>>>> op monitor interval="60s" role="Master" \ >>>>> op monitor interval="59s" role="Slave" \ >>>>> op stop interval="0" timeout="320" \ >>>>> op start interval="0" timeout="240" >>>>> ms AdminClone AdminDrbd \ >>>>> meta master-max="2" master-node-max="1" \ >>>>> clone-max="2" clone-node-max="1" notify="true" >>>>> >>>>> primitive Clvmd lsb:clvmd op monitor interval="30s" >>>>> clone ClvmdClone Clvmd >>>>> colocation Clvmd_With_Admin inf: ClvmdClone AdminClone:Master >>>>> order Admin_Before_Clvmd inf: AdminClone:promote ClvmdClone:start >>>>> >>>>> primitive Gfs2 lsb:gfs2 op monitor interval="30s" >>>>> clone Gfs2Clone Gfs2 >>>>> colocation Gfs2_With_Clvmd inf: Gfs2Clone ClvmdClone >>>>> order Clvmd_Before_Gfs2 inf: ClvmdClone Gfs2Clone >>>>> >>>>> primitive ExportMail ocf:heartbeat:exportfs \ >>>>> op start interval="0" timeout="40" \ >>>>> op stop interval="0" timeout="45" \ >>>>> params clientspec="mail" directory="/mail" fsid="30" >>>>> clone ExportsClone ExportMail >>>>> colocation Exports_With_Gfs2 inf: ExportsClone Gfs2Clone >>>>> order Gfs2_Before_Exports inf: Gfs2Clone ExportsClone > > -- Bill Seligman | Phone: (914) 591-2823 Nevis Labs, Columbia Univ | mailto://[email protected] PO Box 137 | Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
