Hi, On Fri, Jun 05, 2009 at 02:01:19PM +0100, [email protected] wrote: > Hi, > > > Hi, > > > > On Fri, Jun 05, 2009 at 12:06:11PM +0100, [email protected] > wrote: > > > > > > > > Hi, > > > > > > > > On Thu, Jun 04, 2009 at 03:47:03PM +0100, [email protected] > > > wrote: > > > > > Hello, > > > > > this is a little bit off topic but maybe anybody can help me. > The > > > > physical setup of our Debian etch heartbeat cluster looks like > this: > > > > > > > > > > Cluster of two nodes connected with bonding to the network, with > a > > > > crossover connection to each other (interconnect) and with dual > port > > > > Qlogic HBAs to our SAN. The SAN consists of two independent > (physical) > > > > SANs with different VSANs (just to be more human readable) > connected > > > to > > > > some EMC clariions CX3-20f. > > > > > > > > > > We use multipathd, LVM2 and ocfs2. This is the multipath.conf: > > > > > > > > > > defaults { > > > > > rr_min_io 1000 > > > > > polling_interval 2 > > > > > no_path_retry 5 > > > > > user_friendly_names yes > > > > > } > > > > > > > > > > blacklist { > > > > > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > > > > > devnode "^hd[a-z][[0-9]*]" > > > > > devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]" > > > > > device { > > > > > vendor "DGC" > > > > > product "LUNZ" #EMC Clariion management LUN > > > > > } > > > > > device { > > > > > vendor "ATA" #We do not need mutlipathing for > local > > > > drives > > > > > product "*" > > > > > } > > > > > device { > > > > > vendor "AMI" # No multipathing for SUN Virtual > > > devices > > > > > product "*" > > > > > } > > > > > device { > > > > > vendor "HITACHI" # No multipathing for local > scsi > > > disks > > > > > product "H101414SCSUN146G" > > > > > } > > > > > } > > > > > > > > > > devices { > > > > > ## Device attributes for EMC CLARiiON > > > > > device { > > > > > vendor "DGC" > > > > > product "*" > > > > > path_grouping_policy group_by_prio > > > > > getuid_callout "/sbin/scsi_id -g -u -s > > > > /block/%n" > > > > > prio_callout "/sbin/mpath_prio_emc > > > /dev/%n" > > > > > hardware_handler "1 emc" > > > > > features "1 queue_if_no_path" > > > > > no_path_retry fail > > > > > path_checker emc_clariion > > > > > path_selector "round-robin 0" > > > > > failback immediate > > > > > } > > > > > } > > > > > > > > > > multipaths { > > > > > multipath { > > > > > wwid > > > > 3600601603ac511001c7c92fec775dd11 > > > > > alias stosan01_lun070 > > > > > } > > > > > } > > > > > > > > > > > > > > > And we changed the "filter" of lvm2 to not scan /dev/sd* > devices. > > > After > > > > booting up the system multipath -ll looks like this: > > > > > > > > > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 > DGC,RAID 5 > > > > > [size=133G][features=0][hwhandler=1 emc] > > > > > \_ round-robin 0 [prio=2][active] > > > > > \_ 0:0:1:1 sdd 8:48 [active][ready] > > > > > \_ 3:0:1:1 sdh 8:112 [active][ready] > > > > > \_ round-robin 0 [prio=0][enabled] > > > > > \_ 0:0:0:1 sdb 8:16 [active][ready] > > > > > \_ 3:0:0:1 sdf 8:80 [active][ready] > > > > > > > > > > If we now kill one of the connections by disable the zoning on > one > > > of > > > > our SANs this changes to: > > > > > > > > > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 > DGC,RAID 5 > > > > > [size=133G][features=0][hwhandler=1 emc] > > > > > \_ round-robin 0 [prio=1][active] > > > > > \_ 0:0:1:1 sdd 8:48 [active][ready] > > > > > \_ round-robin 0 [prio=0][enabled] > > > > > \_ 0:0:0:1 sdb 8:16 [active][ready] > > > > > > > > > > And /var/log/messages > > > > > > > > > > Jun 4 14:53:58 defr1elcbtd02 kernel: o2net: no longer connected > to > > > node > > > > defr1elcbtd01 (num 0) at 192.168.0.101:7777 > > > > > Jun 4 14:54:01 defr1elcbtd02 kernel: > > > > (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: > > > > waiting 5000ms for notification of death of node 0 > > > > > Jun 4 14:54:16 defr1elcbtd02 last message repeated 3 times > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: > > > Failing > > > > path 8:80. > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: > > > Failing > > > > path 8:112. > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > long > > > > trespass command will be send > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > > honor reservation bit will not be set (default) > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: ioctl: > error > > > adding > > > > target to table > > > > > > > > ? > > > > > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > long > > > > trespass command will be send > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > > honor reservation bit will not be set (default) > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: > > > > (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: > > > > waiting 5000ms for notification of death of node 0 > > > > > Jun 4 14:54:26 defr1elcbtd02 kernel: scsi 1:0:1:1: SCSI error: > > > return > > > > code = 0x00020000 > > > > > > > > This looks like a multipath issue. AFAIK, the SCSI layer > > > > shouldn't perceive a change in path availability. Perhaps better > > > > to post to another list where such problems are discussed. > > > > > > > > Thanks, > > > > > > > > Dejan > > > > > > I will post this to the ocfs2-users mailinglist. Just to note on > Debian > > > lenny we never had this error. Looks like a bug in Debian etch to > me. > > > > Hmm, I doubt that it's the problem of ocfs2, to me it looks more > > like multipathing, i.e. a on some level below. > > > > Thanks, > > > > Dejan > > But if it isn't the ocfs2 cluster service why is it still possible to > write to the mounted SAN volumes?
> > > > > O2CB_HEARTBEAT_THRESHOLD=31 > > > > > O2CB_IDLE_TIMEOUT_MS=30000 > > > > > O2CB_KEEPALIVE_DELAY_MS=2000 > > > > > O2CB_RECONNECT_DELAY_MS=2000 On Mon, Jun 08, 2009 at 10:35:30AM +0100, [email protected] wrote: > The Problem was solved by changing the OCFS2 timeout values to > > O2CB_HEARTBEAT_THRESHOLD=61 > O2CB_IDLE_TIMEOUT_MS=60000 > O2CB_KEEPALIVE_DELAY_MS=4000 > O2CB_RECONNECT_DELAY_MS=4000 OK. So, it seems like ocfs2 needed to relax a bit. I was expecting that multipathing would hide the problem from the upper layer, but obviously there's a glitch in disk connectivity. Thanks, Dejan _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
