> > Hi, > > On Thu, Jun 04, 2009 at 03:47:03PM +0100, [email protected] wrote: > > Hello, > > this is a little bit off topic but maybe anybody can help me. The > physical setup of our Debian etch heartbeat cluster looks like this: > > > > Cluster of two nodes connected with bonding to the network, with a > crossover connection to each other (interconnect) and with dual port > Qlogic HBAs to our SAN. The SAN consists of two independent (physical) > SANs with different VSANs (just to be more human readable) connected to > some EMC clariions CX3-20f. > > > > We use multipathd, LVM2 and ocfs2. This is the multipath.conf: > > > > defaults { > > rr_min_io 1000 > > polling_interval 2 > > no_path_retry 5 > > user_friendly_names yes > > } > > > > blacklist { > > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > > devnode "^hd[a-z][[0-9]*]" > > devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]" > > device { > > vendor "DGC" > > product "LUNZ" #EMC Clariion management LUN > > } > > device { > > vendor "ATA" #We do not need mutlipathing for local > drives > > product "*" > > } > > device { > > vendor "AMI" # No multipathing for SUN Virtual devices > > product "*" > > } > > device { > > vendor "HITACHI" # No multipathing for local scsi disks > > product "H101414SCSUN146G" > > } > > } > > > > devices { > > ## Device attributes for EMC CLARiiON > > device { > > vendor "DGC" > > product "*" > > path_grouping_policy group_by_prio > > getuid_callout "/sbin/scsi_id -g -u -s > /block/%n" > > prio_callout "/sbin/mpath_prio_emc /dev/%n" > > hardware_handler "1 emc" > > features "1 queue_if_no_path" > > no_path_retry fail > > path_checker emc_clariion > > path_selector "round-robin 0" > > failback immediate > > } > > } > > > > multipaths { > > multipath { > > wwid > 3600601603ac511001c7c92fec775dd11 > > alias stosan01_lun070 > > } > > } > > > > > > And we changed the "filter" of lvm2 to not scan /dev/sd* devices. After > booting up the system multipath -ll looks like this: > > > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 DGC,RAID 5 > > [size=133G][features=0][hwhandler=1 emc] > > \_ round-robin 0 [prio=2][active] > > \_ 0:0:1:1 sdd 8:48 [active][ready] > > \_ 3:0:1:1 sdh 8:112 [active][ready] > > \_ round-robin 0 [prio=0][enabled] > > \_ 0:0:0:1 sdb 8:16 [active][ready] > > \_ 3:0:0:1 sdf 8:80 [active][ready] > > > > If we now kill one of the connections by disable the zoning on one of > our SANs this changes to: > > > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 DGC,RAID 5 > > [size=133G][features=0][hwhandler=1 emc] > > \_ round-robin 0 [prio=1][active] > > \_ 0:0:1:1 sdd 8:48 [active][ready] > > \_ round-robin 0 [prio=0][enabled] > > \_ 0:0:0:1 sdb 8:16 [active][ready] > > > > And /var/log/messages > > > > Jun 4 14:53:58 defr1elcbtd02 kernel: o2net: no longer connected to node > defr1elcbtd01 (num 0) at 192.168.0.101:7777 > > Jun 4 14:54:01 defr1elcbtd02 kernel: > (4741,0):dlm_wait_for_node_death:374 5EE89BC01EFC405E9197C198DEEAE678: > waiting 5000ms for notification of death of node 0 > > Jun 4 14:54:16 defr1elcbtd02 last message repeated 3 times > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: Failing > path 8:80. > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: Failing > path 8:112. > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: long > trespass command will be send > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: > honor reservation bit will not be set (default) > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: ioctl: error adding > target to table > > ? > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: long > trespass command will be send > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: > honor reservation bit will not be set (default) > > Jun 4 14:54:21 defr1elcbtd02 kernel: > (4741,0):dlm_wait_for_node_death:374 5EE89BC01EFC405E9197C198DEEAE678: > waiting 5000ms for notification of death of node 0 > > Jun 4 14:54:26 defr1elcbtd02 kernel: scsi 1:0:1:1: SCSI error: return > code = 0x00020000 > > This looks like a multipath issue. AFAIK, the SCSI layer > shouldn't perceive a change in path availability. Perhaps better > to post to another list where such problems are discussed. > > Thanks, > > Dejan
I will post this to the ocfs2-users mailinglist. Just to note on Debian lenny we never had this error. Looks like a bug in Debian etch to me. > > > Jun 4 14:54:26 defr1elcbtd02 kernel: end_request: I/O error, dev sdh, > sector 1672 > > Jun 4 14:54:26 defr1elcbtd02 kernel: device-mapper: multipath emc: > emc_pg_init: sending switch-over command > > Jun 4 14:54:26 defr1elcbtd02 kernel: > (4741,0):dlm_wait_for_node_death:374 5EE89BC01EFC405E9197C198DEEAE678: > waiting 5000ms for notification of death of node 0 > > Jun 4 14:54:57 defr1elcbtd02 last message repeated 6 times > > > > After 60 seconds the kernel panics because of ocfs2 is fencing this > node: > > > > o2quo_make_decision:143 ERROR: fencing this node because it is connected > to a half-quorum of 1 out of 2 nodes which doesn't include the lowest > active node 0 > > > > The o2cb settings are: > > O2CB_HEARTBEAT_THRESHOLD=31 > > O2CB_IDLE_TIMEOUT_MS=30000 > > O2CB_KEEPALIVE_DELAY_MS=2000 > > O2CB_RECONNECT_DELAY_MS=2000 > > > > If we just set the FC Port on one of the switches to shutdown everything > works fine. Qla2xxx detects port down, multipathd detects the missing path > and ocfs2 is not recognizing anything - just working fine like it > should... > > > > Whats going on? Is this a bug? > > > > Regards Florian > > > > > > Florian Engelmann | Unix System and Storage Administrator | IT Services > Germany | BT (Germany) GmbH & Co. oHG | Tel:+49 (0)89 2600 8173 | Mob:+49 > (0)176 1073 1081 | Fax:+49 (0)89 2600 4955 | Email: > [email protected] | Sitz und Registergericht der oHG: M?nchen, HRA > 77639 | Pers?nlich haftende Gesellschafterin der oHG: BT Deutschland GmbH > | Sitz und Registergericht: M?nchen, HRB 132307 | Gesch?ftsf?hrung: > Karsten Lereuth, Joachim Piroth, Dr. Sebastian Brandis, Martin Glock, Neil > Smith, Judith Sasse | www.bt.com/globalservices > > > > Diese E-Mail enth?lt Informationen von BT (Germany) GmbH & Co. oHG. > Diese sind m?glicherweise vertraulich und ausschlie?lich f?r den > Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere > Personen als den Adressaten ist untersagt. Sollten Sie nicht der f?r diese > E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, > Vervielf?ltigung oder Weitergabe untersagt. Sollten Sie diese > elektronische Nachricht irrt?mlicherweise erhalten haben, so informieren > Sie uns bitte unverz?glich telefonisch oder per E-Mail (Kontaktdaten > obenstehend). Das E-Mail-System wird von BT (Germany) GmbH & Co. oHG im > Rahmen der gesetzlichen Bestimmungen laufend ?berwacht, insbesondere um > einen st?rungsfreien Betrieb zu gew?hrleisten. Auch die vom System > ?bertragenen Nachrichten werden im Rahmen der gesetzlichen Bestimmungen > durch BT (Germany) GmbH & Co. oHG ?berwacht und k?nnen aufgezeichnet oder > ausgewertet werden. > > This electronic message contains information from BT (Germany) GmbH & > Co. oHG which may be privileged or confidential. The information is > intended to be for the use of the individual(s) or entity named above. If > you are not the intended recipient be aware that any disclosure, copying, > distribution or use of the contents of this information is prohibited. If > you have received this electronic message in error, please notify us by > telephone or e-mail (to the numbers or address above) immediately. > Activity and use of the BT (Germany) GmbH & Co. oHG e-mail system is > monitored to secure its effective operation and for other lawful business > purposes. Communications using this system will also be monitored and may > be recorded to secure effective operation and for other lawful business > purposes. > > > > > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
