The Problem was solved by changing the OCFS2 timeout values to O2CB_HEARTBEAT_THRESHOLD=61 O2CB_IDLE_TIMEOUT_MS=60000 O2CB_KEEPALIVE_DELAY_MS=4000 O2CB_RECONNECT_DELAY_MS=4000
> > Hi, > > > Hi, > > > > On Fri, Jun 05, 2009 at 12:06:11PM +0100, [email protected] > wrote: > > > > > > > > Hi, > > > > > > > > On Thu, Jun 04, 2009 at 03:47:03PM +0100, [email protected] > > > wrote: > > > > > Hello, > > > > > this is a little bit off topic but maybe anybody can help me. > The > > > > physical setup of our Debian etch heartbeat cluster looks like > this: > > > > > > > > > > Cluster of two nodes connected with bonding to the network, with > a > > > > crossover connection to each other (interconnect) and with dual > port > > > > Qlogic HBAs to our SAN. The SAN consists of two independent > (physical) > > > > SANs with different VSANs (just to be more human readable) > connected > > > to > > > > some EMC clariions CX3-20f. > > > > > > > > > > We use multipathd, LVM2 and ocfs2. This is the multipath.conf: > > > > > > > > > > defaults { > > > > > rr_min_io 1000 > > > > > polling_interval 2 > > > > > no_path_retry 5 > > > > > user_friendly_names yes > > > > > } > > > > > > > > > > blacklist { > > > > > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > > > > > devnode "^hd[a-z][[0-9]*]" > > > > > devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]" > > > > > device { > > > > > vendor "DGC" > > > > > product "LUNZ" #EMC Clariion management LUN > > > > > } > > > > > device { > > > > > vendor "ATA" #We do not need mutlipathing for > local > > > > drives > > > > > product "*" > > > > > } > > > > > device { > > > > > vendor "AMI" # No multipathing for SUN Virtual > > > devices > > > > > product "*" > > > > > } > > > > > device { > > > > > vendor "HITACHI" # No multipathing for local > scsi > > > disks > > > > > product "H101414SCSUN146G" > > > > > } > > > > > } > > > > > > > > > > devices { > > > > > ## Device attributes for EMC CLARiiON > > > > > device { > > > > > vendor "DGC" > > > > > product "*" > > > > > path_grouping_policy group_by_prio > > > > > getuid_callout "/sbin/scsi_id -g -u -s > > > > /block/%n" > > > > > prio_callout "/sbin/mpath_prio_emc > > > /dev/%n" > > > > > hardware_handler "1 emc" > > > > > features "1 queue_if_no_path" > > > > > no_path_retry fail > > > > > path_checker emc_clariion > > > > > path_selector "round-robin 0" > > > > > failback immediate > > > > > } > > > > > } > > > > > > > > > > multipaths { > > > > > multipath { > > > > > wwid > > > > 3600601603ac511001c7c92fec775dd11 > > > > > alias stosan01_lun070 > > > > > } > > > > > } > > > > > > > > > > > > > > > And we changed the "filter" of lvm2 to not scan /dev/sd* > devices. > > > After > > > > booting up the system multipath -ll looks like this: > > > > > > > > > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 > DGC,RAID 5 > > > > > [size=133G][features=0][hwhandler=1 emc] > > > > > \_ round-robin 0 [prio=2][active] > > > > > \_ 0:0:1:1 sdd 8:48 [active][ready] > > > > > \_ 3:0:1:1 sdh 8:112 [active][ready] > > > > > \_ round-robin 0 [prio=0][enabled] > > > > > \_ 0:0:0:1 sdb 8:16 [active][ready] > > > > > \_ 3:0:0:1 sdf 8:80 [active][ready] > > > > > > > > > > If we now kill one of the connections by disable the zoning on > one > > > of > > > > our SANs this changes to: > > > > > > > > > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 > DGC,RAID 5 > > > > > [size=133G][features=0][hwhandler=1 emc] > > > > > \_ round-robin 0 [prio=1][active] > > > > > \_ 0:0:1:1 sdd 8:48 [active][ready] > > > > > \_ round-robin 0 [prio=0][enabled] > > > > > \_ 0:0:0:1 sdb 8:16 [active][ready] > > > > > > > > > > And /var/log/messages > > > > > > > > > > Jun 4 14:53:58 defr1elcbtd02 kernel: o2net: no longer connected > to > > > node > > > > defr1elcbtd01 (num 0) at 192.168.0.101:7777 > > > > > Jun 4 14:54:01 defr1elcbtd02 kernel: > > > > (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: > > > > waiting 5000ms for notification of death of node 0 > > > > > Jun 4 14:54:16 defr1elcbtd02 last message repeated 3 times > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: > > > Failing > > > > path 8:80. > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: > > > Failing > > > > path 8:112. > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > long > > > > trespass command will be send > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > > honor reservation bit will not be set (default) > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: ioctl: > error > > > adding > > > > target to table > > > > > > > > ? > > > > > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > long > > > > trespass command will be send > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > > honor reservation bit will not be set (default) > > > > > Jun 4 14:54:21 defr1elcbtd02 kernel: > > > > (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: > > > > waiting 5000ms for notification of death of node 0 > > > > > Jun 4 14:54:26 defr1elcbtd02 kernel: scsi 1:0:1:1: SCSI error: > > > return > > > > code = 0x00020000 > > > > > > > > This looks like a multipath issue. AFAIK, the SCSI layer > > > > shouldn't perceive a change in path availability. Perhaps better > > > > to post to another list where such problems are discussed. > > > > > > > > Thanks, > > > > > > > > Dejan > > > > > > I will post this to the ocfs2-users mailinglist. Just to note on > Debian > > > lenny we never had this error. Looks like a bug in Debian etch to > me. > > > > Hmm, I doubt that it's the problem of ocfs2, to me it looks more > > like multipathing, i.e. a on some level below. > > > > Thanks, > > > > Dejan > > But if it isn't the ocfs2 cluster service why is it still possible to > write to the mounted SAN volumes? > > > > > > > > > > > > > > > > > > Jun 4 14:54:26 defr1elcbtd02 kernel: end_request: I/O error, > dev > > > sdh, > > > > sector 1672 > > > > > Jun 4 14:54:26 defr1elcbtd02 kernel: device-mapper: multipath > emc: > > > > emc_pg_init: sending switch-over command > > > > > Jun 4 14:54:26 defr1elcbtd02 kernel: > > > > (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: > > > > waiting 5000ms for notification of death of node 0 > > > > > Jun 4 14:54:57 defr1elcbtd02 last message repeated 6 times > > > > > > > > > > After 60 seconds the kernel panics because of ocfs2 is fencing > this > > > > node: > > > > > > > > > > o2quo_make_decision:143 ERROR: fencing this node because it is > > > connected > > > > to a half-quorum of 1 out of 2 nodes which doesn't include the > lowest > > > > active node 0 > > > > > > > > > > The o2cb settings are: > > > > > O2CB_HEARTBEAT_THRESHOLD=31 > > > > > O2CB_IDLE_TIMEOUT_MS=30000 > > > > > O2CB_KEEPALIVE_DELAY_MS=2000 > > > > > O2CB_RECONNECT_DELAY_MS=2000 > > > > > > > > > > If we just set the FC Port on one of the switches to shutdown > > > everything > > > > works fine. Qla2xxx detects port down, multipathd detects the > missing > > > path > > > > and ocfs2 is not recognizing anything - just working fine like it > > > > should... > > > > > > > > > > Whats going on? Is this a bug? > > > > > > > > > > Regards Florian > > > > > > > > > > > > > > > Florian Engelmann | Unix System and Storage Administrator | IT > > > Services > > > > Germany | BT (Germany) GmbH & Co. oHG | Tel:+49 (0)89 2600 8173 | > > > Mob:+49 > > > > (0)176 1073 1081 | Fax:+49 (0)89 2600 4955 | Email: > > > > [email protected] | Sitz und Registergericht der oHG: > M?nchen, > > > HRA > > > > 77639 | Pers?nlich haftende Gesellschafterin der oHG: BT > Deutschland > > > GmbH > > > > | Sitz und Registergericht: M?nchen, HRB 132307 | > Gesch?ftsf?hrung: > > > > Karsten Lereuth, Joachim Piroth, Dr. Sebastian Brandis, Martin > Glock, > > > Neil > > > > Smith, Judith Sasse | www.bt.com/globalservices > > > > > > > > > > Diese E-Mail enth?lt Informationen von BT (Germany) GmbH & Co. > oHG. > > > > Diese sind m?glicherweise vertraulich und ausschlie?lich f?r den > > > > Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch > andere > > > > Personen als den Adressaten ist untersagt. Sollten Sie nicht der > f?r > > > diese > > > > E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, > > > > Vervielf?ltigung oder Weitergabe untersagt. Sollten Sie diese > > > > elektronische Nachricht irrt?mlicherweise erhalten haben, so > > > informieren > > > > Sie uns bitte unverz?glich telefonisch oder per E-Mail > (Kontaktdaten > > > > obenstehend). Das E-Mail-System wird von BT (Germany) GmbH & Co. > oHG > > > im > > > > Rahmen der gesetzlichen Bestimmungen laufend ?berwacht, > insbesondere > > > um > > > > einen st?rungsfreien Betrieb zu gew?hrleisten. Auch die vom System > > > > ?bertragenen Nachrichten werden im Rahmen der gesetzlichen > > > Bestimmungen > > > > durch BT (Germany) GmbH & Co. oHG ?berwacht und k?nnen > aufgezeichnet > > > oder > > > > ausgewertet werden. > > > > > This electronic message contains information from BT (Germany) > GmbH > > > & > > > > Co. oHG which may be privileged or confidential. The information > is > > > > intended to be for the use of the individual(s) or entity named > above. > > > If > > > > you are not the intended recipient be aware that any disclosure, > > > copying, > > > > distribution or use of the contents of this information is > prohibited. > > > If > > > > you have received this electronic message in error, please notify > us > > > by > > > > telephone or e-mail (to the numbers or address above) immediately. > > > > Activity and use of the BT (Germany) GmbH & Co. oHG e-mail system > is > > > > monitored to secure its effective operation and for other lawful > > > business > > > > purposes. Communications using this system will also be monitored > and > > > may > > > > be recorded to secure effective operation and for other lawful > > > business > > > > purposes. > > > > > > > > > > > > > > > _______________________________________________ > > > > > Linux-HA mailing list > > > > > [email protected] > > > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > > > See also: http://linux-ha.org/ReportingProblems > > > > _______________________________________________ > > > > Linux-HA mailing list > > > > [email protected] > > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > > See also: http://linux-ha.org/ReportingProblems > > > _______________________________________________ > > > Linux-HA mailing list > > > [email protected] > > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > > See also: http://linux-ha.org/ReportingProblems > > _______________________________________________ > > Linux-HA mailing list > > [email protected] > > http://lists.linux-ha.org/mailman/listinfo/linux-ha > > See also: http://linux-ha.org/ReportingProblems > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
