[Bug 1020436] Re: Cannot read superblock after FC multipath failover
To assist in further debug, stop the multipath daemon, then run it in the foreground with max verbosity, while running it under strace. Log strace to file and the foreground stdout and stderr to another file. Post the results here. Also include your multipath.conf, lvm.conf, output of dmsetup table, and 'echo show config | multipathd -k'. While the system is in the failure mode, in running ps aux do you see a series of kpartx processes in uninterruptable state? I presume you've configured your lvm.conf blacklist correctly? Thanks. https://help.ubuntu.com/12.04/serverguide/multipath-devices.html #multipath-devices-in-logical-volumes ** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Peter Petrakis (peter-petrakis) ** Changed in: multipath-tools (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1013724] Re: Setting "prio const" in multipath.conf has no effect
Please provide the entire multipath.conf file, in the meanwhile try stopping the daemon, perform a table flush (multipath -F) , then simply run multipath -v4 (no daemon) and observe the results. ** Changed in: multipath-tools (Ubuntu) Status: New => Incomplete ** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Peter Petrakis (peter-petrakis) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1013724 Title: Setting "prio const" in multipath.conf has no effect To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1013724/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 803554] Re: upgrading to multipath 0.4.9 presents incompatible config file changes
This has been resolved by updating the documentation in the serverguide. https://help.ubuntu.com/12.04/serverguide/device-mapper- multipathing.html#multipath-new-and-changed-features ** Changed in: multipath-tools (Ubuntu) Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/803554 Title: upgrading to multipath 0.4.9 presents incompatible config file changes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/803554/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 927540] Re: multipath ignores blacklist in multipath.conf
multipath supports POSIX regexp: libmultipath/regex.c so you do need
the "." to inform the wildcard what it's matching against e.g. (1+ more
characters).
There are also examples in the multipath.conf.defaults file that provide
examples
of regexp like:
# device {
# vendor "HP"
# product "LOGICAL VOLUME.*"
# getuid_callout "/lib/udev/scsi_id --whitelisted
--device=/dev/%n"
and
# device {
# vendor "DGC"
# product ".*"
# product_blacklist "LUNZ"
If you want to wildcard an entire field, leaving out the dot works, but when you
start matching within strings, you need to specify what it is you're matching.
#blacklist {
# devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# devnode "^hd[a-z]"
# devnode "^dcssblk[0-9]*"
#}
You could try a dramatically simplified multipath.conf like this:
blacklist {
device {
vendor "*"
product "*"
}
}
blacklist_exceptions {
device {
vendor "NETAPP"
product "LUN"
}
}
The continue configuring your SAN under a new devices{ device {} } section.
In my case I just take the defaults. to prove to yourself that this works,
stop multipath server, flush all paths (multipath -F), and then change
the config file to include only the blacklist (exclude everything), run
multipath -v4 and observe.
= paths list =
uuid hcildev dev_t pri dm_st chk_st vend/prod
3600508b1001cb2fce7d75a0d5bee1ea1 3:0:0:1 sda 8:0 1 undef faulty HP,LOGICA
360a98000534b504d6834654d53793373 2:0:0:0 sdb 8:16 4 undef ready NETAPP,LU
360a98000534b504d6834654d53793373 5:0:0:0 sdc 8:32 4 undef ready NETAPP,LU
360a98000534b504d6834654d53793373 2:0:1:0 sdd 8:48 1 undef ready NETAPP,LU
360a98000534b504d6834654d53793373 5:0:1:0 sde 8:64 1 undef ready NETAPP,LU
Jul 09 15:16:07 | sda: (HP:LOGICAL VOLUME) vendor/product blacklisted
Jul 09 15:16:07 | sdb: (NETAPP:LUN) vendor/product blacklisted
Jul 09 15:16:07 | sdc: (NETAPP:LUN) vendor/product blacklisted
Jul 09 15:16:07 | sdd: (NETAPP:LUN) vendor/product blacklisted
Jul 09 15:16:07 | sde: (NETAPP:LUN) vendor/product blacklisted
root@nashira:~# lsscsi
[0:0:0:0]cd/dvd hp DVD A DS8A5LH 1HE3 /dev/sr0
[2:0:0:0]diskNETAPP LUN 8020 /dev/sdb
[2:0:1:0]diskNETAPP LUN 8020 /dev/sdd
[3:0:0:0]storage HP P410i3.66 -
[3:0:0:1]diskHP LOGICAL VOLUME 3.66 /dev/sda
[5:0:0:0]diskNETAPP LUN 8020 /dev/sdc
[5:0:1:0]diskNETAPP LUN 8020 /dev/sde
So it indeed blacklisted everything. and multipath -ll produces nothing.
Once I add my SAN to the blacklist exception block everything functions
as expected.
create: 360a98000534b504d6834654d53793373 undef NETAPP,LUN
size=16G features='1 queue_if_no_path' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=4 status=undef
| |- 2:0:0:0 sdb 8:16 undef ready running
| `- 5:0:0:0 sdc 8:32 undef ready running
`-+- policy='round-robin 0' prio=1 status=undef
|- 2:0:1:0 sdd 8:48 undef ready running
`- 5:0:1:0 sde 8:64 undef ready running
root@nashira:~# multipath -ll
Error: : Inappropriate ioctl for device
cciss TUR failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
360a98000534b504d6834654d53793373 dm-0 NETAPP,LUN
size=16G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=4 status=active
| |- 2:0:0:0 sdb 8:16 active ready running
| `- 5:0:0:0 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
|- 2:0:1:0 sdd 8:48 active ready running
`- 5:0:1:0 sde 8:64 active ready running
** Changed in: multipath-tools (Ubuntu)
Status: Confirmed => Incomplete
** Changed in: multipath-tools (Ubuntu)
Assignee: (unassigned) => Peter Petrakis (peter-petrakis)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/927540
Title:
multipath ignores blacklist in multipath.conf
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/927540/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
In examining the ioctl interface to device mapper (kernel), I have not found any architecture that assures that when a device mapper device has been marked for deletion, that this information is relayed back to userspace so it can block on this deletion. An example will clarify things. In SD we have a state machine the details the teardown of a SCSI device. Deletion is "latched" in two FSM states, SDEV_CANCEL and SDEV_DEL. The first state tells everyone that this device is on it's way out and to stop trying to use it, DEL is the final state where callbacks start the kobj destruction process. In this model, it's easy to determine that a device is being deleted by examining the state variable in sysfs. It's of little consequence here as SD devices are not deterministic, the same name need not be used, if sdd was being deleted, and another hotplug create event was generated, SD would hand you sde. Deterministic names are left to udev. In the device mapper case, the WWID is a deterministic and unique name, there is no substitute. Once the device has been marked for deletion, there's no readily apparent way to verify that it has in fact been marked for deletion. dev_status is the closest we can get, and the flags it provides are not clear enough to determine that a device is being deleted. Therefore, the mechanism does not exist to even poll for the destruction of a dm name, as we can't even determine that the dm name is being destroyed. We couldn't wait for deletion if we wanted to. http://lxr.linux.no/linux+v3.4.4/drivers/md/dm-ioctl.c#L656 So this really isn't a bug, it's an example of DM's current design. If we want to address the issue then we'll have to propose an enhanced tear down protocol, and then update any device mapper consumers to use this protocol. There might be a way to infer a deletion state from a combination of flags, but that will take a concentrated effort to determine. This issue is a synthetic fault at best, there's no real impact to the user. ** Changed in: multipath-tools (Ubuntu) Status: Triaged => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/585027 Title: Race condition with dmsetup causes 'map already present' messages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
Should really be won't fix but I apparently can't set it. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/585027 Title: Race condition with dmsetup causes 'map already present' messages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
@Alasdair
It's buried in the description:
"We were attempting variants of 'while true; do sudo multipath -F; sudo
multipath -v4 ; done' to create a set of udev add/remove events and
noticed that shortly after starting, no output appeared within the
'udevadm monitor' command."
So it appears that multipath is sending a DM ioctl to create a map with the
same wwid,
that "should" have been completely removed at the exit of 'multipath -F'.
There's
no dmsetup command executing out of band unless there's some udev rule
misbehaving.
The udev rules managing dmsetup are /lib/udev/rules.d/55-dm.rules
I'm not familiar with the cookie feature you mentioned, but we appear to have
some of the
capability here. I haven't compared it to upstream yet.
"55-dm.rules"
# Decode udev control flags and set environment variables appropriately.
# These flags are encoded in DM_COOKIE variable that was introduced in
# kernel version 2.6.31. Therefore, we can use this feature with
# kernels >= 2.6.31 only.
ENV{DM_COOKIE}=="?*", IMPORT{program}="/sbin/dmsetup udevflags $env{DM_COOKIE}"
ENV{DM_COOKIE}=="?*", RUN+="/sbin/dmsetup udevcomplete $env{DM_COOKIE}"
and compared to our packaged source in precise:
<12:29:34>multipath-tools-0.4.9$ grep -R cookie .
./kpartx/devmapper.c:dm_simplecmd (int task, const char *name, int no_flush,
uint32_t *cookie) {
./kpartx/devmapper.c: if (udev_wait_flag && !dm_task_set_cookie(dmt, cookie,
0))
./kpartx/devmapper.c: mode_t mode, uid_t uid, gid_t gid, uint32_t *cookie)
{
./kpartx/devmapper.c: if (task == DM_DEVICE_CREATE &&
!dm_task_set_cookie(dmt, cookie, 0))
./kpartx/kpartx.c: uint32_t cookie = 0;
./kpartx/kpartx.c:0, &cookie)) {
./kpartx/kpartx.c: buf.st_gid,
&cookie)) {
./kpartx/kpartx.c:1, &cookie)) {
./kpartx/kpartx.c:&cookie);
./kpartx/kpartx.c:
&cookie);
./kpartx/kpartx.c: dm_udev_wait(cookie);
./libmultipath/config.h:uint32_t cookie;
./libmultipath/devmapper.c: if (udev_wait_flag && !dm_task_set_cookie(dmt,
&conf->cookie, 0))
./libmultipath/devmapper.c: !dm_task_set_cookie(dmt, &conf->cookie, 0))
./libmultipath/devmapper.c: if (!dm_task_set_cookie(dmt, &conf->cookie, 0))
./multipath/main.c: dm_udev_wait(conf->cookie);
This cookie mechanism is news to me, I'll take a closer look, thanks.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/585027
Title:
Race condition with dmsetup causes 'map already present' messages
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
So when this bug was first reported we were using multipath 0.4.8, it doesn't appear to have this cookie mechanism. 0.4.9 does (oneiric and higher); I have been unable to reproduce the issue with 0.4.9. Going back through the logs... """ >From 8d63b33d0996e141a2451df552b062b908db15bc Mon Sep 17 00:00:00 2001 From: Benjamin Marzinski Date: Mon, 17 May 2010 14:03:38 -0500 Subject: [PATCH] multipath: add udev sync support. device-mapper in now able to synchronize operations through udev. This patch allows multipath and kpartx to make use of this feature. If kpartx is run with "-s", it waits for the partitions to be created before returning. Multipath will now always wait for the devices to be created before returning. This feature requires dm_task_set_cookie() which was finalized in device-mapper version 1.2.38 """ Appears to be where cookie support was initially added. I'll retest with 0.4.8 to verify that the fault still exists (probably) and take it from there. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/585027 Title: Race condition with dmsetup causes 'map already present' messages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
Your blacklist definition is wrong. You provided.
"""
blacklist {
wwid "3600605b0039afe20ff54052e7d38"
vendor "SMART"
product "SMART"
}
defaults {
user_friendly_names yes
}
"""
wwid cites an entire device, vendor & product cite a different device and must
be
included in device { } block. If you look at the "show config" output, you'll
note
that your blacklisted device by vendor/product isn't there.
Change to:
"""
blacklist {
wwid "3600605b0039afe20ff54052e7d38"
device {
vendor "SMART"
product "SMART"
}
}
defaults {
user_friendly_names yes
}
"""
update your initrd (multipath.conf is copied there) and reboot.
Also, it would be nice to see a list of all your storage devices. I prefer
lsscsi, e.g.
root@nashira:~# lsscsi
[0:0:0:0]cd/dvd hp DVD A DS8A5LH 1HE3 /dev/sr0
[2:0:0:0]diskNETAPP LUN 8020 /dev/sdb
[2:0:1:0]diskNETAPP LUN 8020 /dev/sdd
[3:0:0:0]storage HP P410i3.66 -
[3:0:0:1]diskHP LOGICAL VOLUME 3.66 /dev/sda
[5:0:0:0]diskNETAPP LUN 8020 /dev/sdc
[5:0:1:0]diskNETAPP LUN 8020 /dev/sde
Finally, since this is FC, how is it zoned? Detailed port level topology please.
Thanks.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1020436
Title:
Cannot read superblock after FC multipath failover
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1013724] Re: Setting "prio const" in multipath.conf has no effect
If your device is mounted, you won't be able to flush it, flush == destroy. OK. so it's not just you, I can recreate this. Coincidentally, I also have an HP RAID controller and see the same warning message. This device *is blacklisted* but apparently it's still getting probed. Shouldn't be causing any problems, though blacklisted should mean what it says. ** Changed in: multipath-tools (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1013724 Title: Setting "prio const" in multipath.conf has no effect To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1013724/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
** Changed in: multipath-tools (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
Thanks for all the feedback, there's quite a bit of information here to sort through. In the meanwhile, it would be an interesting data point to see how well this array functions under multipath 0.4.9, which is what Precise, the new LTS uses. This can be accomplished by using apt pinning. http://askubuntu.com/questions/103320/install-packages-from-newer- release-without-building-apt-pinning/103338#103338 In your case: $ sudo tee /etc/apt/sources.list.d/precise.list << EOF deb http://us.archive.ubuntu.com/ubuntu/ precise main restricted deb-src http://us.archive.ubuntu.com/ubuntu/ precise main restricted deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted deb-src http://us.archive.ubuntu.com/ubuntu/ precise-updates main restricted EOF $ sudo tee /etc/apt/preferences << EOF Package: multipath-tools* Pin: release n=precise Pin-Priority: 990 Package: * Pin: release n=lucid Pin-Priority: 900 Package: * Pin: release o=Ubuntu Pin-Priority: -10 EOF $ sudo apt-get update $ sudo apt-get upgrade Should update multipath-tools and boot to 0.4.9, if not then $ sudo apt-get install -t precise multipath-tools multipath-tools-boot This should unjam you while we triage the data and will provide a valuable data point to aid in backporting if necessary. Note that the config syntax has changed slightly. https://help.ubuntu.com/12.04/serverguide/device-mapper- multipathing.html#multipath-new-and-changed-features It's ok to leave both directives in the file multpath simply globs keywords and uses the ones that "fit". Remember to update your initrd after changing the config file. I assume you have multiple SAN clients, please keep one at 0.4.8 so we can continue to debug this. Thanks. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
Alrighty. [OK] lvm.conf device blacklist [OK] multipath.conf, sans fix I already pointed out You can ignore that FW blurb, it's just a script that shouldn't be so noisy. failover logs *look good* bug-1020436$ grep fail_path multipathd_stdout.log | sort -u Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath0 NF fail_path 8:112 Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath0 NF fail_path 8:160 Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath1 NF fail_path 8:128 Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath1 NF fail_path 8:176 Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath2 NF fail_path 8:144 Jul 10 15:08:25 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath2 NF fail_path 8:192 # 3 mins go by... oh and several devices are failed "twice" # Jul 10 15:10:19 devices start to return Jul 10 15:11:50 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath0 NF fail_path 8:16 Jul 10 15:11:50 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath0 NF fail_path 8:64 Jul 10 15:11:50 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath1 NF fail_path 8:32 Jul 10 15:11:50 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath1 NF fail_path 8:80 Jul 10 15:11:50 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath2 NF fail_path 8:48 Jul 10 15:11:50 | libdevmapper: ioctl/libdm-iface.c(1740): dm message mpath2 NF fail_path 8:96 and according to udev, note that I re-sequenced the grep and grouped by mpathX. bug-1020436$ grep -A4 PATH_FAILED multipathd_stdout.log -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=1 Jul 10 15:08:25 | DM_PATH=8:112 Jul 10 15:08:25 | DM_NR_VALID_PATHS=3 Jul 10 15:08:25 | DM_NAME=mpath0 -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=2 Jul 10 15:08:25 | DM_PATH=8:160 Jul 10 15:08:25 | DM_NR_VALID_PATHS=2 Jul 10 15:08:25 | DM_NAME=mpath0 -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=3 Jul 10 15:08:25 | DM_PATH=8:160 Jul 10 15:08:25 | DM_NR_VALID_PATHS=2 Jul 10 15:08:25 | DM_NAME=mpath0 -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=1 Jul 10 15:08:25 | DM_PATH=8:128 Jul 10 15:08:25 | DM_NR_VALID_PATHS=3 Jul 10 15:08:25 | DM_NAME=mpath1 -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=2 Jul 10 15:08:25 | DM_PATH=8:176 Jul 10 15:08:25 | DM_NR_VALID_PATHS=2 Jul 10 15:08:25 | DM_NAME=mpath1 -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=3 Jul 10 15:08:25 | DM_PATH=8:176 Jul 10 15:08:25 | DM_NR_VALID_PATHS=2 Jul 10 15:08:25 | DM_NAME=mpath1 Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=1 Jul 10 15:08:25 | DM_PATH=8:144 Jul 10 15:08:25 | DM_NR_VALID_PATHS=3 Jul 10 15:08:25 | DM_NAME=mpath2 -- Jul 10 15:08:25 | DM_ACTION=PATH_FAILED Jul 10 15:08:25 | DM_SEQNUM=2 Jul 10 15:08:25 | DM_PATH=8:192 Jul 10 15:08:25 | DM_NR_VALID_PATHS=2 Jul 10 15:08:25 | DM_NAME=mpath2 This all looks OK, you can see the paths count down and stop at the right count. The way multibus works is a lot like network bonding link aggregation, io is simply steered down the remaining paths. What might have happened is the LVs backed by this mpath device somehow got deactivated and that's why the filesystem can't write back. Try this for me please. Shutdown those paths and keep them down, then after about a minute, run. # lvs -o lv_attr We should see "w" and "a" for every volume, if not, that's the problem. Also, while in the "failed state", see if you can do a dd read from one of the mpaths? like dd if=/dev/mapper/mpath0 of=/dev/null count=10. If that works, try the same thing with a random sd device. I don't see anything amiss in the strace. The SG ioctls look OK and the ioctls made to DM make sense. We want to determine at which level that sending block ios is hung up. ** Changed in: multipath-tools (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
Retested on precise base and downgraded to multipath-tools 0.4.8-14ubuntu4 on generic 3.2.0.27.29. Ran unit test for several hours, no fault found. Intend to downgrade to lucid kernel and retest, hopefully it pops then, save me the trouble of installing lucid somewhere. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/585027 Title: Race condition with dmsetup causes 'map already present' messages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
Using the lucid kernel, I was able to reproduce, though it took a lot longer than original report implied, easily over an hour, probably more than 3 I wasn't watching it the entire time. multipath-tools 0.4.8-14ubuntu4.10.04.2 kernel 2.6.32-41-generic udevadm monitor simply stops responding. I cannot remove the dm maps, they say "busy/in use". They are not mounted or used by anyone. Using the same kernel and multipath 0.4.9-3ubuntu5 I was unable to reproduce the issue. So this older kernel helps expose the problem. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/585027 Title: Race condition with dmsetup causes 'map already present' messages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 585027] Re: Race condition with dmsetup causes 'map already present' messages
** Changed in: multipath-tools (Ubuntu) Status: Invalid => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/585027 Title: Race condition with dmsetup causes 'map already present' messages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/585027/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
Hmm, we've got some counter indicators here.
lvs claims that the volumes are active. But the probe itself
is showing problems reading the volumes.
XFS is telling us that it cannot write it's journal to disk.
[1039812.311433] Filesystem "dm-4": Log I/O Error Detected. Shutting down
filesystem: dm-4
"fs/xfs/xfs_rw.c"
90 /*
91 * Force a shutdown of the filesystem instantly while keeping
92 * the filesystem consistent. We don't do an unmount here; just shutdown
93 * the shop, make sure that absolutely nothing persistent happens to
94 * this filesystem after this point.
95 */
96 void
97 xfs_do_force_shutdown(
...
127 if (flags & SHUTDOWN_CORRUPT_INCORE) {
128 xfs_cmn_err(XFS_PTAG_SHUTDOWN_CORRUPT, CE_ALERT, mp,
129 "Corruption of in-memory data detected. Shutting down filesystem: %s",
130 mp->m_fsname);
131 if (XFS_ERRLEVEL_HIGH <= xfs_error_level) {
132 xfs_stack_trace();
133 }
134 } else if (!(flags & SHUTDOWN_FORCE_UMOUNT)) {
135 if (logerror) {
136 xfs_cmn_err(XFS_PTAG_SHUTDOWN_LOGERROR, CE_ALERT,
mp,
137 "Log I/O Error Detected. Shutting down filesystem: %s",
138 mp->m_fsname);
The logs conveniently tell us where it was called from too.
1009 void
1010 xlog_iodone(xfs_buf_t *bp)
1011 {
...
1036 /*
1037 * Race to shutdown the filesystem if we see an error.
1038 */
1039 if (XFS_TEST_ERROR((XFS_BUF_GETERROR(bp)), l->l_mp,
1040 XFS_ERRTAG_IODONE_IOERR, XFS_RANDOM_IODONE_IOERR))
{
1041 xfs_ioerror_alert("xlog_iodone", l->l_mp, bp,
XFS_BUF_ADDR(bp));
1042 XFS_BUF_STALE(bp);
1043 xfs_force_shutdown(l->l_mp, SHUTDOWN_LOG_IO_ERROR);
1044 /*
1045 * This flag will be propagated to the trans-committed
1046 * callback routines to let them know that the log-commit
1047 * didn't succeed.
1048 */
1049 aborted = XFS_LI_ABORTED;
I assume dm-4 is the LV that XFS is mounted on, did you run the dd test
on that?
I'm starting to wonder if the LVM device filter is lying to us, after failover,
something
changes which misrepresents the LV and then XFS bails out.
If you can perform that DD for every PV that backs dm-4 successfully then
there's
something wrong with the DM map for those LVs after failover occurs.
OK, what I need from you now is a before and after (same fault injection
method) of:
0) ls -lR /dev/ > dev_major_minor.log
1) lvs -o lv_attr
2) pvdisplay -vvv
3) lvdisplay -vvv
4) dmsetup table -v
5) "dd test" on all block devices: lv, mp, sd
6) dmesg output
Please attach this as a single tarball, that has a timestamp in the filename
and has a directory structure of:
foo.tgz
before/
after/
If this all checks out, then what's probably happening is the when
multipath begins the failover process, there's enough of a delay that
XFS simply bails out early before IO is ready to be sent down the
remaining paths. group by priority may perform better here and is
something you can test.
I looked at the XFS mount arguments and didn't find anything that would
make it more lenient in these situations.
If you can manage it, a LUN formatted with ext3 under these circumstances
would help in ruling out whether the filesystem is part of the problem.
Thanks.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1020436
Title:
Cannot read superblock after FC multipath failover
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
scratch the ext3 test, I just re-read the description. "I've replicated this behaviour using both xfs and ext4 filesystems, on multiple different luns presented to the server." -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
It doesn't look like your lvm.conf filter is working. (pvdisplay) /dev/sdb: block size is 4096 bytes /dev/sdb: lvm2 label detected Ignoring duplicate PV RLOhdDURbD7uK2La3MDK2olkP0BF2Tu7 on /dev/sdb - using dm /dev/mapper/mpath0 Closed /dev/sdb Opened /dev/sdc RO O_DIRECT the sd names shouldn't even be seen. from the output of vgscan -vvv you should see something like this: /dev/ram15: No label detected Closed /dev/ram15 /dev/sdc: Skipping (regex) /dev/sdc1: Skipping (regex) Change your filter to what's documented in the multipath guide. https://help.ubuntu.com/12.04/serverguide/multipath-devices.html #multipath-devices-in-logical-volumes filter = [ "r/block/", "r/disk/", "r/sd.*/", "a/.*/" ] Run vgscan -vvv after to verify, those pvdisplay errors should also go away. If you want it to be extra quiet, you can filter out ram devices too, here's mine on an hp server with the following filter. filter = [ "r/block/", "r/ram.*/", "r/disk/", "r/cciss/", "r/sd.*/", "a/.*/" ] The thing to know about sd names is that they are not deterministic, ever, so to keep sda as part of your vg set you'll need to determine it's unique udev name and filter that in. Looking back at your original filter, I can see that it's wrong now. # By default we accept every block device: filter = [ "a/.*/", "r|/dev/sd[b-z]|" ] It should be # By default we accept every block device: filter = [ "a/.*/", "r/sd.[b-z]/" ] Again, if you hotplug sda, or reverse scan your busses, this ceases filter accurately, use the unique udev names. Beyond that, everything looks good, all block devices are responding, that are left, after failover. I tested failover using lvm + xfs using multipath 0.4.8 with both multibus and priority grouping, I could not reproduce the issue. Load test was iozone -a in a loop. I just realized, you didn't describe what load these were under during the failure, or how many failovers are required to reach the fault. You're dmtables look sane. -0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:64 1000 8:16 1000 8:96 1000 8:160 1000 +0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 8:16 1000 You can tell that it's all in one group via the single round-robin directive, which marks the beginning of all paths in that group, clearly you went from 4 -> 2. The fact that the map is updated at all means multipathd did it's job. multipathd's main job is the patch checking, and managing the policy we define for those paths by updating the dm tables as needed. DM then simply operates on what it has for a table definition and keeps chugging along until that table changes. back to the original multipathd output you provide DM_STATE=active and DM_SUSPENDED=0, that map was never deactivated. I tried pushing my queue up to 1000 from 128 for rr_min_io, no change. If it's not the lvm filter then it's something else very subtle. Unless you're comfortable with upgrading to multipath 0.4.9 via apt pinning (done this before with lucid, very stable, and better) as I described in comment #16, the only way we'll get to the bottom of this is if we're hands on; which necessitates an Ubuntu Advantage agreement. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
** Changed in: multipath-tools (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1020436] Re: Cannot read superblock after FC multipath failover
Could you please file a new bug against LVM/lucid (just add a tag 'lucid') detailing this behavior? Thanks. I've never been a fan of LVM's filtering mechanism, then again the source is there, I could do something about it :) What you described here: """ However as soon as I use something like: filter = [ "a|.*|", "r|loop[0-9]|" ] Then I don't get any filtering at all... except that defining _only_ the removal filters, per: filter = [ "r|loop[0-9]+|" ] .. and other filters of that ilk DO work as expected -- so long as I remove the "accept-all" regex. """ Sounds like a bug to me. The whole point of this filter-in, filter-out mechanism is to perform operations that exclude everything by default, and then allow only certain devices in. Just like you would configure a firewall. As for multipath aliasing, it happens more often than I would like, which is why I made such a point to include it into the official documentation. Once you start stacking block devices it can get complicated very quickly. We should probably expand the docs to include udev alias examples, as configurations like yours are the second most common e.g. disk 1 not part of VG but everything else is. Glad we could get you sorted. I really appreciate the level of detail and responsiveness you were able to provide. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1020436 Title: Cannot read superblock after FC multipath failover To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1030040] Re: multipath-tools create duplicate device
Hi Vincent, Could you please provide? 1) any lvm config file or other "stacked" block device configurations 2) if lvm: vgscan -vvv 2>&1 | tee > vgscan.log 3) multipath.conf 4) echo 'show config' | multipathd -k > multipathd-runtime-config.log 5) dmsetup table -v > dmsetup.log 6) multipath -v4 -ll > multipath.log 7) system logs put that together into a tarball with a datetime stamp and attach it to the bug. To assist in debugging. Please stop multipathd, flush all paths and then only run multipath client so we can observe how the paths are created. 0) service multipath-tools stop 1) multipath -F # check that paths have been deleted 2) multipath -v4 > multipath-create-all-no-daemon.log 3) dmsetup table -v > dmsetup-no-daemon.log 3) multipath -ll > multipath-normal-listing.log attach all this in another tarball appropriately labelled and attach. BTW, is your hardware handler actually installed? lsmod | grep scsi_dh Help me understand why your SD names have 'lapped" to use an additional character, are you performing a lot of hotplug or are there that many disks? an lsscsi -l output would be nice. Thanks. ** Changed in: multipath-tools (Ubuntu) Status: New => Incomplete ** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Peter Petrakis (peter-petrakis) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1030040 Title: multipath-tools create duplicate device To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1030040/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 927540] Re: multipath ignores blacklist in multipath.conf
@gheeke Did you try the alternate config file style I proposed in comment #5? Thanks. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/927540 Title: multipath ignores blacklist in multipath.conf To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/927540/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1030040] Re: multipath-tools create duplicate device
Hmm, A reproduction case is going to be necessary, there are some
weird things going on in syslog.2 which I'm unfamiliar with. Unfortunately,
I don't have access to a symmetrix either. I really need to have multipathd
running
in the foreground, with full logging to a file, to catch what went down.
>From 7/27/12 00:38:48 -- 00:38:58 I just see lines and lines of this:
Jul 27 00:38:58 Linux51 multipathd: mpath12: sdm - emc_clariion_checker:
Logical Unit is unbound or LUNZ
For about every block device, this messages means the path is down. It's safety
against issuing IO
to a LUNZ which will block forever.
libmultipath/checkers/emc_clariion.c
if ( /* LUN should at least be bound somewhere and not be LUNZ */
sense_buffer[4] == 0x00) {
MSG(c, "emc_clariion_checker: Logical Unit is unbound "
"or LUNZ");
return PATH_DOWN;
}
After a few filters...
Jul 27 05:45:36 Linux51 kernel: [188724.191711] scsi 6:0:0:0: Direct-Access
DGC LUNZ 0532 PQ: 0 ANSI: 4
Jul 27 05:45:36 Linux51 kernel: [188724.193584] scsi 6:0:1:0: Direct-Access
DGC LUNZ 0532 PQ: 0 ANSI: 4
Jul 27 05:45:43 Linux51 kernel: [188730.940803] scsi 5:0:0:0: Direct-Access
DGC LUNZ 0532 PQ: 0 ANSI: 4
Jul 27 05:45:43 Linux51 kernel: [188730.942590] scsi 5:0:1:0: Direct-Access
DGC LUNZ 0532 PQ: 0 ANSI: 4
ok, that's 4, but there were dozens in the logs, which means the rest
were unbound.
Then there's this,
Jul 27 00:39:49 Linux51 udevd[26799]: timeout '/sbin/blkid -o udev -p
/dev/dm-53'
Jul 27 00:39:49 Linux51 udevd[26794]: timeout '/sbin/blkid -o udev -p
/dev/dm-52'
Jul 27 00:39:50 Linux51 udevd[26848]: timeout: killing '/sbin/blkid -o udev -p
/dev/dm-66' [26890]
Jul 27 00:39:50 Linux51 udevd[26857]: timeout: killing '/sbin/blkid -o udev -p
/dev/dm-60' [26899]
...
Something is off, if multipath did create two maps to the same device I suspect
it to be a victim
of these path availability issues.
<11:37:45>multipath_conf$ zcat syslog.2.gz | grep unbound | sort -u | wc -l
43055
That can't be right.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1030040
Title:
multipath-tools create duplicate device
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1030040/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1030040] Re: multipath-tools create duplicate device
** Changed in: multipath-tools (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1030040 Title: multipath-tools create duplicate device To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1030040/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Peter Petrakis (peter-petrakis) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
@Vincent Could you please attach the syslog associated with this fault? How reproducible is this? Also I noticed that your lvm.conf is a little broken. scan = [ "/dev" ] ... filter = [ "r/block/", "r/disk/", "r|/dev/sd.*|", "a/.*/" ] Since the scan starts in /dev, that's the root of the search, specifying "r|/dev/sd.*|" won't match anything, instead you want "r|sd.*|". vgscan -vvv will show you the regexp working. https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1020436/comments/23 ** Changed in: multipath-tools (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
** Changed in: multipath-tools (Ubuntu) Status: Incomplete => In Progress -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1030040] Re: multipath-tools create duplicate device
@Vincent: I thought you asked for this case to be closed? Did you mean to update another bug? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1030040 Title: multipath-tools create duplicate device To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1030040/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
@Vincent I have a test package for you to try. Add this ppa, https://launchpad.net/~peter-petrakis/+archive/storage The install incantation should be: apt-get install multipath-tools=0.4.9-3ubuntu5+lp1032250dbg1 It includes a fix to the state checker which *should* eliminate the sysfs coherency issue. Based on RH BZ 680480: https://bugzilla.redhat.com/show_bug.cgi?id=680480 ** Bug watch added: Red Hat Bugzilla #680480 https://bugzilla.redhat.com/show_bug.cgi?id=680480 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
So you restarted multipath-tools or rebooted before testing again? I was afraid of this, the RH patch isn't even upstream and was applied against a different code base. There may be additional patches that have a cumulative effect that we're missing. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
Have you verified that the rdac driver is loaded? Also please account for the contents of /etc/initramfs-tools/modules, scsi_dh_rdac must be loaded at boot time to be discovered correctly. ** Changed in: multipath-tools (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1057054 Title: poor performance after upgrade to Precise To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
That's never how it works. multipath has no kernel module loading ability. Would make a nice feature though. I admit that discovering "which dh is the necessary one" is a bit arcane and not well documented anywhere. The best practice here is to load the necessary device handler into your initrd so it's attached at the same time the SD devices are initially discovered. You still need to have this module loaded to provision additional luns at runtime and not have their performance plummet. Also, I discovered a typo in the multipath documentation. https://bugs.launchpad.net/serverguide/+bug/1057071 Had that worked to begin with, you would have never encountered this issue in the first place. Closing this issue as "Invalid" as it's not a bug. Thanks for the report. ** Changed in: multipath-tools (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1057054 Title: poor performance after upgrade to Precise To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
There is no feature, that has ever existed, that has the capacity to *at runtime* examine all attached disks, and cross reference their SCSI INQUIRY data to a table of available device handlers. That table does not exist, if it did, it would be miserable to maintain. I checked the udev rules and initramfs scripts from lucid -> precise. We never loaded dh modules automatically. The multipath C code has no facility to modprobe or insmod anything. So the only logical conclusion left is that the module was loaded without your knowledge, which means your configuration as it was would never survive a reboot. If that's not true, and you can reproduce that, I would be interested to see it. However, even if I had the answer, that doesn't completely make up for a complete lack of vendor participation in qualifying your SAN with our operating [1]. We cannot be expected to regression test every SAN in creation and rely on users like you (or vendors) to test and stay engaged. Please contact your vendor expressing support for official Ubuntu support for your SAN multipath-tools is supported by the Community, not Canonical, I volunteer to maintain it. That multipath section in the server guide? I wrote it with the next precise LTS as the deadline, months of effort. multipath as a whole is light years better than it was in lucid, or ever for that matter (many helped). I'm not disagreeing with you that things are missing and there's certainly room for improvement. You've pointed out several issues, like the dialog box,man page etc, that's all good stuff, please file a separate bug for each so we can track them. It's simply a matter of triage and bandwidth, a good multipath bug can soak weeks of time, so configuration polish like you mentioned falls to the way side. However, that sort of work is low hanging fruit, and doesn't require kernel storage engineer with years of experience to accomplish. Contributions are most certainly welcome. FYI, there really isn't a hard spec for multipath.conf, it actually functions a lot like YAML where keywords are globbed, the values integrated and override the defaults. There's no one place in the code where you can go and discover "this is how config works", it's scattered everywhere which makes creating regression tests prohibitive if not practically impossible. 1. The implication is that multipath may have changed so dramatically from 0.4.8 to 0.4.9 that the scsi_dh_rdac driver may not have been as necessary. There's no way we could have caught that on code review, testing was required. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1057054 Title: poor performance after upgrade to Precise To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
Hmm, that's interesting, does that mean that after reboot scsi_dh_rdac is
loaded?
Please verify.
Yes, it is available with the lucid kernel. Also note that if you were to
reference
your vendor documentation, it would probably recommend that you load
rdac driver (been around for a long time actually).
NOTE: multipath is basically the same on every distro, so instructions for RH
provided by your vendor are just as good for Ubuntu etc. However kernels
change frequently which is why vendors and others need to be involved.
Let's assume that rdac is never loaded and life is good on lucid. That leaves
multipath itself and the Linux kernel, both of which have jumped dramatically
between lucid and precise. Since there's no real regression tests for HW SAN,
and
the vendors aren't pitching in, it's easily possible that something weird
like this could occur.
It's my opinion that if you're using ALUA, it's up to you to determine whether
an additional device handler is necessary. What may have happened is multipath
in lucid was biasing the primary storage controller and forcing a trespass
unbenounced to you. This would have made ALUA irrelevant and ping ponged your
luns behind the RAID for a period of time, it also means you weren't using both
your
storage processors like you intended. That would have been a bonafide bug in
lucid.
It's likely that it was rectified in precise considering the outcome. Going
back and
finding it is an academic exercise as no matter what I find, it'll probably
break
lucid. RDAC must be loaded.
[ALUA architecture example]
http://virtualgeek.typepad.com/virtual_geek/2009/09/a-couple-important-alua-and-srm-notes.html
As for the kernel, it does what it's told, something with that horrendous an
impact would have likely impacted your non-san disks as well, if it affected
only
one that would be double weird :) So it likely comes down to how the IO was
queued
to begin with, and multipathd is responsible for that.
Between the two MP versions, your SAN actually got a codified config, meaning
you don't need to provide your own if you don't want to, it's built in. 0.4.8
didn't
have this.
[libmultipath/hwtable.c]
{
/* DELL MD3000 */
.vendor= "DELL",
.product = "MD3000",
.getuid= DEFAULT_GETUID,
.features = "2 pg_init_retries 50",
.hwhandler = "1 rdac",
.selector = DEFAULT_SELECTOR,
.pgpolicy = GROUP_BY_PRIO,
.pgfailback= -FAILBACK_IMMEDIATE,
.rr_weight = RR_WEIGHT_NONE,
.no_path_retry = 15,
.minio = DEFAULT_MINIO,
.checker_name = RDAC,
.prio_name = PRIO_RDAC,
.prio_args = NULL,
},
Your config is overridding any member you defined, the rest are coming through,
like
minio.
BTW you might wish to double check exactly what your SAN can do. Active/Active
isn't what it used to be and is really "dual active".
http://gestaltit.com/all/tech/storage/stephen/multipath-activepassive-
dual-active-activeactive/
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1057054
Title:
poor performance after upgrade to Precise
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
Then we might have a distro bug here, which is weird as I've done hundreds of
SAN installs with lucid and have had to manage the scsi_dh modules everytime.
So on your lucid system, /etc/initramfs-tools/modules should be empty
except for the commented out examples.
The next thing to check is the initram disks themselves, assuming there's no
directives in the previous config file, the presence of the scsi_dh kos in
the initrd would indicate that they were be globbed in by another
initramfs helper.
Actually...
root@nashira:~# zcat /boot/initrd.img-3.2.0-26-generic | cpio -it | grep
scsi_dh
lib/modules/3.2.0-26-generic/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko
lib/modules/3.2.0-26-generic/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko
lib/modules/3.2.0-26-generic/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko
lib/modules/3.2.0-26-generic/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko
81384 blocks
root@nashira:~# zcat /boot/initrd.img-2.6.32-41-generic | cpio -it | grep
scsi_dh
lib/modules/2.6.32-41-generic/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko
lib/modules/2.6.32-41-generic/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko
lib/modules/2.6.32-41-generic/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko
lib/modules/2.6.32-41-generic/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko
64508 blocks
OK, I'm surprised :)
In my 2.6.32 initrd I find:
root@nashira:~/2.6.32# cat conf/modules
scsi_dh_alua
which I suppose prompts my device handler to be installed. Which is driven
by 'modules' and 'modules.d' in /usr/share/initramfs-tools. So the modules
apparently have always been there, thanks to this hook script.
/usr/share/initramfs-tools/hook-functions
scsi)
copy_modules_dir kernel/drivers/scsi
for x in mptfc mptsas mptscsih mptspi zfcp; do
manual_add_modules "${x}"
done
;;
Which is how the scsi_dh_* kos got on the initramfs, but that doesn't
explain how it was loaded. That directive had to come from somewhere,
so it's either already in your /usr/share/initramfs-tools/modules|modules.d
or it's in your /etc/initramfs-tools/modules.d|modules and you missed it.
*something* has to be prompting it's inclusion.
What's the output of, as root?
grep -Rl scsi_dh /etc /usr/share/initramfs-tools/
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1057054
Title:
poor performance after upgrade to Precise
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
There's nothing in any of the priority checkers except scsi cmds. See for yourself. bzr+ssh://bazaar.launchpad.net/+branch/ubuntu/lucid/multipath-tools/ path_priority/pp_rdac/pp_rdac.c -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1057054 Title: poor performance after upgrade to Precise To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1057054] Re: poor performance after upgrade to Precise
Then the only possible actors left are the actual initramdisk contents e.g. zcat | cpio -id and examine all the scripts (init, conf/modules) to determine how it could be loaded. The absence of the module reference in all of /etc and /usr/share/initramfs-tools/ tell me that whatever is loading that module is doing so as a side effect or an administration artifact e.g. someone wrote a udev rule and forgot. Another possible source and this is also way out in left field is if modules.dep was compromised and scsi_sh_rdac was added as dependency of another module and thus loaded indirectly. root@nashira:/lib/modules/2.6.32-41-generic# grep scsi_dh modules* modules.builtin:kernel/drivers/scsi/device_handler/scsi_dh.ko modules.dep:kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko: modules.dep:kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko: modules.dep:kernel/drivers/scsi/device_handler/scsi_dh_emc.ko: modules.dep:kernel/drivers/scsi/device_handler/scsi_dh_alua.ko: Binary file modules.dep.bin matches modules.order:kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko modules.order:kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko modules.order:kernel/drivers/scsi/device_handler/scsi_dh_emc.ko modules.order:kernel/drivers/scsi/device_handler/scsi_dh_alua.ko root@nashira:/lib/modules/2.6.32-41-generic# vim modules.dep root@nashira:/lib/modules/2.6.32-41-generic# vim modules.builtin Fine here. concerning boot probe, scsi discovery is asymmetric, there's no expectation of order. Performance tuning is where I get off, I also don't know much about iSCSI transport, though yeah, jumbo frames it probably wise. If you're seeing those messages before the file system is mounted then the actors are definitely *in the ramdisk*, you just need to find them. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1057054 Title: poor performance after upgrade to Precise To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1057054/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 927540] Re: multipath ignores blacklist in multipath.conf
That devnode line is actually part of the default in memory config, see libmultipath/blacklist.c#setup_default_blist. So try dropping that. Just try and keep you multpath.conf as simple as possible, to even *one* directive, test and record, then iterate until you start to see it break. It's best to stop multipath-tools altogether and just "multipath -F && multipath -v4" until you find the breakage. Record the logs the entire time using redirection or 'script' and attach your results to this issue. Thank you. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/927540 Title: multipath ignores blacklist in multipath.conf To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/927540/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 927540] Re: multipath ignores blacklist in multipath.conf
That's actually a good point Lukasz, one that I assume folks get by reading the multipath guide first. https://help.ubuntu.com/12.04/serverguide/multipath-devices.html Whenever you use blacklists, in either lvm or multipath, you must update the initramfs so those config files are copied to the ramdisk, otherwise you can get a race that makes it look like your blacklist is having no effect. # update-initramfs -u -k all should do it. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/927540 Title: multipath ignores blacklist in multipath.conf To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/927540/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Blueprint servercloud-r-kdump-tool] Enable kdump mechanism from kdump-tool instead of kexec-tools
Blueprint changed by Peter Petrakis: Work items set to: Work items: Integrate netdump support: TODO netdump charm to aggregate dumps in a cloud: TODO -- Enable kdump mechanism from kdump-tool instead of kexec-tools https://blueprints.launchpad.net/ubuntu/+spec/servercloud-r-kdump-tool -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1073948] [NEW] CryptographicSourceVerification wiki has syntax errors
Public bug reported: https://juju.ubuntu.com/CryptographicSourceVerification The snippet that makes use of the charm-helpers ppa is off, it won't function as a copy/paste. The PPA name may be plural but the contents of it are not. This is correct: apt-get -qqy charm-helper-sh # loading helper library . /usr/share/charm-helper/sh/net.sh Also worthy of note, add-apt-repository should probably have a -y switch as it prompts "are you sure" for PPA addition. I'd fix it myself but the wiki isn't editable. ** Affects: juju (Ubuntu) Importance: Undecided Status: New ** Tags: documentation -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to juju in Ubuntu. https://bugs.launchpad.net/bugs/1073948 Title: CryptographicSourceVerification wiki has syntax errors To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/juju/+bug/1073948/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 927540] Re: multipath ignores blacklist in multipath.conf
Considering the time past and that I'm convinced that this is a user configuration issue (Support) that should be addressed by comment #11, I'm closing this issue as Invalid. ** Changed in: multipath-tools (Ubuntu) Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/927540 Title: multipath ignores blacklist in multipath.conf To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/927540/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
What I really need is a crashdump. I'm simply short on time to reproduce this myself. https://wiki.ubuntu.com/Kernel/CrashdumpRecipe Configure sysctl to panic on oops, that should do it. This stack section is interesting. Dec 9 09:18:27 ealxs00195 kernel: [1726974.078795] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 ... Dec 9 09:18:27 ealxs00195 kernel: [1726974.091694] [] __scsi_remove_target+0xd4/0xf0 Dec 9 09:18:27 ealxs00195 kernel: [1726974.091696] [] scsi_remove_target+0xc1/0xe0 Dec 9 09:18:27 ealxs00195 kernel: [1726974.091701] [] fc_starget_delete+0x26/0x30 [scsi_transport_fc] Dec 9 09:18:27 ealxs00195 kernel: [1726974.091706] [] process_one_work+0x11a/0x480 It's required to be able to sleep in a workq context, we're grabbing spinlocks here, which when done quickly is fine, but holding them too long and well, you see this. I doubt this is the root cause, something else is likely holding the locks due to an error handling condition and the removal threads are just victims. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
I should have been more specific, please set kernel.hung_task_panic as well, which should trigger on the "blocked more than N sec" events. Also, if you still have the system in this state. Does ps | aux show a series of kpartx processes blocked? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
hmm, try this. kernel.panic_on_oops = 0 kernel.softlockup_panic = 0 kernel.unknown_nmi_panic = 0 kernel.panic_on_unrecovered_nmi = 0 kernel.panic_on_io_nmi = 0 kernel.hung_task_panic = 0 Make sure these are all set to 1 in /etc/sysctl.conf and then run sysctl -p Also, lets configure ftrace, see if we get lucky and can catch the deadlock as it happens. [as root] cd /sys/kernel/debug/tracing echo function > current_tracer echo 16000 > buffer_size_kb echo "scsi_*" >> set_ftrace_filter echo "fc_*" >> set_ftrace_filter When the kernel crashes, I'll be able to retrieve this buffer from the dump Should none of the dump triggers work, then just watch it and trigger it yourself. # echo c > /proc/sysrq-trigger The hung multipath -v0 processes are usually spawned on behalf of udev events. Before you begin the test, please record the current dm map (dmsetup table -v) and the output of (lsscsi -lv) so I have a point of reference. Thanks. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
Hi Ronald, Sorry I haven't been timely, this is the best I can do with community level support If kdump isn't launching even in the most trivial case then you have to start from zero. is crashkernel even configured? * grep crash /proc/crashkernel How much memory do you have, could you assign more memory to the crash kernel? * http://lxr.linux.no/linux+v3.7.1/Documentation/kdump/kdump.txt#L270 * 256MB would be preferable Can you even kexec at all? * kexec -p # loads the panic kernel, man kexec If you boot your system with maxcpus=1 (I think that's it) and pretend you're a uniprocessor system, will kexec load? Can you attach a serial console to your machine and post the output? In /etc/init.d/kdump # Append kdump_needed for initramfs to know what to do, and add # maxcpus=1 to keep things sane. APPEND="$APPEND kdump_needed maxcpus=1 irqpoll reset_devices" Start adjusting these variables, like remove 'reset_devices', reload the kexec kernel (service kdump restart), and systematically remove variables (except kdump_needed) noting the change in the kernel output. Is this an enterprise server with an NMI button? If you configure "panic on nmi" pressing that button, that will definitely change the base variables used to launch kexec. Folks thought Stratus was a bit overkill, having a complete mirror of CPU/Memory operating in lockstep for HA. The nice thing about it is if the primary ever did crash, we would literally hold that unit in stasis, reboot on the other unit, and reap the dump from it's preserved memory, works 100% and automatic. Be nice to have right about now. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
Correction: is crashkernel even configured? * grep crash /proc/cmdline -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
^5! Make sure you enable ftrace per: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/comments/16 That'll save me a lot of work, should crash with a nice long call graph and the CPUs caught executing the code. Well done! -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
@Ronald
First, please attach http://www.rmoesbergen.nl/vmcore-crash.tgz to the bug,
launchpad
can handle it just fine. Also, this is going to take awhile. We're off all next
week so don't
expect any movement on this until early-mid Jan. Feel free to ping me if I
forget.
Also, at what time did your testing start? I'm seeing this everywhere almost
immediately
emc: ALUA failover mode detected
Could you also illustrate what the steady state target distribution
should be?
I see targets like this:
sd 3:0:0:0: [sdb] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB)
in the minority compared to
sd 3:0:0:1: [sdc] 419430400 512-byte logical blocks: (214 GB/200 GiB)
Wondering if your SAN is misreporting READ CAPACITY.
The dump looks good. Immediately I can tell you that all the scsi hosts
are still RUNNING and not in error handling. It looks like I'll have examine
the scsi target states and the dm tables.
So there are these stuck processes
crash> ps | grep UN
1530 2 0 880415ef9700 UN 0.0 0 0 [jbd2/dm-1-8]
2180 2 1 88040613ae00 UN 0.0 0 0 [flush-252:1]
4739 1 2 880418e7 UN 5.8 16426520 1029488 mysqld
Which adds up, you can't write back.
This also looks really suspicious.
[62856.457650] end_request: I/O error, dev sdf, sector 21272960
[62856.457966] device-mapper: multipath: Failing path 8:80.
[62856.462495] scsi 3:0:0:0: emc: Detached
[62856.462730] device-mapper: multipath: Failing path 8:80.
[62856.462798] sd 4:0:0:0: emc: ALUA failover mode detected
[62856.462806] sd 4:0:0:0: emc: at SP A Port 0 (owned, default SP A)
# sketchy
[62856.462814] device-mapper: multipath: Could not failover the device: Handler
scsi_dh_emc Error 15.
# it looks like it's retrying
[63122.241178] sd 3:0:1:0: [sdf] Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
[63122.241185] sd 3:0:1:0: [sdf] CDB: Write(10): 2a 00 01 44 b4 d8 00 00 20 00
[63122.241198] end_request: I/O error, dev sdf, sector 21279960
[63122.241513] device-mapper: multipath: Failing path 8:80.
[63122.244865] scsi 3:0:0:0: emc: Detached
[63122.245045] sd 4:0:0:0: emc: ALUA failover mode detected
[63122.245053] sd 4:0:0:0: emc: at SP A Port 0 (owned, default SP A)
# sketchy
[63122.245062] device-mapper: multipath: Could not failover the device: Handler
scsi_dh_emc Error 15.
...
which comes from: [drivers/md/dm-mpath.c]
case SCSI_DH_NOSYS:
if (!m->hw_handler_name) {
errors = 0;
break;
}
DMERR("Could not failover the device: Handler scsi_dh_%s "
"Error %d.", m->hw_handler_name, errors);
/*
* Fail path for now, so we do not ping pong
*/
fail_path(pgpath);
break;
Hey, was this intentional?
[0.018792] Hardware name: ProLiant DL380p Gen8
[0.018794] Your BIOS is broken and requested that x2apic be disabled
[0.018795] This will leave your machine vulnerable to irq-injection attacks
[0.018796] Use 'intremap=no_x2apic_optout' to override BIOS request
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1032550
Title:
[multipath] failed to get sysfs information
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1135453] Re: open-iscsi +mpio with multipathd init script order errors
** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Peter Petrakis (peter-petrakis) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1135453 Title: open-iscsi +mpio with multipathd init script order errors To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1135453/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1135453] Re: open-iscsi +mpio with multipathd init script order errors
This might be self inflicted. First, we have really good multipath documentation now. You're better off following our docs first and then supplementing it with the vendor docs if it doesn't work. Many popular SANs have built-in support. https://help.ubuntu.com/12.04/serverguide/dm-multipath-chapter.html HP MSA's are already included in the built-in DB, your config file is likely not needed so lets start by eliminating that as a variable by following these directions. https://help.ubuntu.com/12.04/serverguide/multipath-setting-up-dm- multipath.html#multipath-setup-overview plus your path alias, and, # update-initramfs -u -k all To copy the multipath.conf to the ramdisk. Make all these changes, reboot, and before you shutdown again grab: * sudo dmsetup table * sudo udisks --dump * sudo -i; echo 'show config' | multipathd -k and attach to the bug. Thanks. I'm hoping you have a newer MSA and this is just a mp jumbled config. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1135453 Title: open-iscsi +mpio with multipathd init script order errors To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1135453/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1135453] Re: open-iscsi +mpio with multipathd init script order errors
** Changed in: multipath-tools (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1135453 Title: open-iscsi +mpio with multipathd init script order errors To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1135453/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1178721] Re: multipathd fails to create mappings when multipath.conf is present
https://help.ubuntu.com/12.10/serverguide/multipath-devices.html You cannot have a default LVM config co-exist with multipath. They will race to grab the SD device for map creation and whoever gets there first wins; they both use device mapper. LVM is configured using UDEV so it's the first pass, multipath in run by legacy rc script. You must configure LVM to blacklist all SD devices. Then, you must configure LVM to use allow the disks necessary for the root volume using the UDEV names e.g. /dev/disk/by-id SD names are never deterministic, sorry, that's what udev us for. When the multipath volume is not root, you can easily test the config per the documentation. Any time a change is made to either multipath.conf or lvm.conf, the initrd must be updated; also in the documentation. Multipath usually runs last, so if you didn't push the lvm blacklist to the ramdisk, by the time multipath runs there's nothing left for it to attach to. Your "virtual multipath" is bogus, there's nothing that will arbitrate write ordering, the block names aren't deterministic, and they never can be because they don't have make/model/serial# which udev uses to create unique names. You could easily "multipath" your root disk + a MP leg and be left with swiss cheese based on scan ordering or hotplug events. If you remove LVM from the stack and follow the directions for properly propagating multipath to the ramdisk, it will work. Start small, then add an additional layer of complexity. There does not exist a tool which integrates the deployment of LVM + multipath anywhere, you have to understand both and configure by hand. It is challenging to deploy an LVM config that co-exits with multipath. Good luck. ** Changed in: multipath-tools (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1178721 Title: multipathd fails to create mappings when multipath.conf is present To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1178721/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1178721] Re: multipathd fails to create mappings when multipath.conf is present
** Changed in: multipath-tools (Ubuntu) Status: Triaged => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1178721 Title: multipathd fails to create mappings when multipath.conf is present To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1178721/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 1178721] Re: multipathd fails to create mappings when multipath.conf is present
At this point I would seriously consider obtaining commercial support. This is clearly a support issue and not a bug and has been closed as invalid. LVM could have easier to grok filtering but it does work, you just have to get creative by changing the scan directory. For example: UNTESTED scan = [ "/dev/mapper"] filter = [ "a|^/dev/mapper/mpath.*|", "r/.*/" ] pvscan -vvv is your friend. Concerning your other SAN question, please seek professional support or seek forums (google) where SAN admins gather. We can't know how to configure *every single SAN on the market*. Again, LVM & multipath are distro agnostic, *any other distro doc will do*. e.g. https://access.redhat.com/site/documentation/en- US/Red_Hat_Enterprise_Linux/5/html/Logical_Volume_Manager_Administration/lvm_filters.html On 06/10/2013 03:33 PM, Jernej Jakob wrote: > Thank you for your help. I wasn't aware of the limitation > (incompatibility) caused by co-deploying LVM and multipath, it was my > assumption that multipath would be configured to take precedence over > LVM for having PVs on multipath block devices, as having it the other > way around, as in multiple paths over LVM logical volumes would not make > sense. I can assume that this would be a pretty common scenario, as for > example, you may have one volume mapped from SAN over FC on which the > tools that LVM offers would be valuable. > > It would seem that a simple check at boot time, checking whether LVM and > multipath are co-deployed, starting multipath first, letting it create > paths and then adding those found to LVM's filter. > > I understand however that this is in no way common usage and requires a > skillful enough administrator to be aware of the possible scenarios > caused. > > Regarding the multipath config itself, this is the default config, > functionally the same as the one proposed for MSA2012fc by HP. > The virtual one is of course bogus... not to be used, just to demonstrate the > failure mode. The actual servers will have a dual-port FC HBA with each port > going to separate zones on a switch, with itself being connected with 2 > cables each to the 2 controllers (4 in total) of the SAN, so there will be 2 > paths for each volume. This is required for this SAN in order to have > redundancy for controller failure. > > I just tried this with the physical server, and it still doesn't work as > it should. In lvm.conf, blacklisted all the sd* devices and allowed only > /dev/disk/by-id. Or should even the id's of the MP volumes be > blacklisted, only allowing the one single PV's id? > -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1178721 Title: multipathd fails to create mappings when multipath.conf is present To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1178721/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1197448] Re: having multipath-tools installed prevents mounting of USB card reader
multipath-tools will grab any devices that are not excluded by the filtering directives in multipath.conf. It's a lot like lvm.conf. The operating system has no idea what these devices will be used for, you must direct it explicitly. https://help.ubuntu.com/lts/serverguide/multipath-dm-multipath-config- file.html#multipath-config-blacklist closing, support issue. ** Changed in: multipath-tools (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1197448 Title: having multipath-tools installed prevents mounting of USB card reader To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1197448/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1217933] Re: update-grub fails to detect other md OS
Why do you think this is a multipath problem and not an mdadm problem? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1217933 Title: update-grub fails to detect other md OS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1217933/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1217933] Re: update-grub fails to detect other md OS
not a multipath issue ** Package changed: multipath-tools (Ubuntu) => grub2 (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1217933 Title: update-grub fails to detect other md OS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1217933/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1217933] Re: update-grub fails to detect other md OS
Right. multipath is for SANs, not RAIDs. Seeing that both of your arrays are up and running however implies a grub configuration problem. Moving... -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1217933 Title: update-grub fails to detect other md OS To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1217933/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 716659] Re: Root filesystem goes "Read only" after installing multipath-tools on Lucid
>From examining the output and can see why the blacklist was
defined in this manner, and how to correct it.
Also, since the luns are reporting themselves as DGC, specifying
a configuration for SYMMETRIX just won't work. Clarion and
Symmetrix are completely different animals from what I gather.
Attached is a new multipath.conf (rename it correctly before
moving to etc) that contains what I believe to be the configuration
they're trying to achieve. Unfortunately, this is array is so configurable,
all this effort could be a waste if it's not setup correctly.
Should this config not work, try deleting the 'devices' section
entirely and reboot. Multipath carries a configuration for DGC
already that might work.
# echo 'show config' | multipathd -k
...
device {
vendor DGC
product .*
product_blacklist LUNZ
path_grouping_policy group_by_prio
path_checker emc_clariion
features 1 queue_if_no_path
hardware_handler 1 emc
prio_callout /sbin/mpath_prio_emc /dev/%n
failback immediate
no_path_retry 60
}
Failing that I would suggest that the customer make the array work with
powerpath first, so we know the HW is configured correctly, and then
we can try again with multipath.
** Attachment added: "Experimental EMC Clarion config"
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/716659/+attachment/1878975/+files/multipath-v1.conf
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in ubuntu.
https://bugs.launchpad.net/bugs/716659
Title:
Root filesystem goes "Read only" after installing multipath-tools on
Lucid
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 716659] Re: Root filesystem goes "Read only" after installing multipath-tools on Lucid
** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Ubuntu Storage Development Team (ubuntu-storage-dev) ** Summary changed: - Root filesystem goes "Read only" after installing multipath-tools on Lucid + [EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/716659 Title: [EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 788326] Re: Ubuntu 10.04 LTS guest did not boot after install
** Visibility changed to: Private -- You received this bug notification because you are a member of Ubuntu Server Team, which is a direct subscriber. https://bugs.launchpad.net/bugs/788326 Title: Ubuntu 10.04 LTS guest did not boot after install -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
We had another incident of this fault, but with an EMC Clariion CX4-120.
After eliminating LVM, and moving to a default MP config. It wasn't
until we removed the "change" rule to the 95-multipath.tools script
that the udev error messages ceased. I'm convinced that this
approach for the boot scripts is broken now and requires additional
scrutiny.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
@John, I've got a test for you.
- stop multipathd: service multipath-tools stop
- monitor udev events as root: udevadm monitor
There should be no new change events.
then in a separate terminal, as root run:
/sbin/mpath_prio_netapp /dev/sdb (or any netapp device)
do you see a change event each time you run the cmd above?
Thanks.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
Excellent, you're still interested. I've got a fix for you to try.
Attached is a custom version of mpath_prio_netapp that should
not generate any more change events for you. Please post
for comparison that this new version returns the same data
as the old one.
move the old mpath_prio_netapp to /sbin/mpath_prio_netapp.old
cp the new one into sbin.
post the results of the following script
(adjust device range accordingly)
for i in sd[a-z]; do /sbin/mpath_prio_netapp.old $i; done
for i in sd[a-z]; do /sbin/mpath_prio_netapp $i; done
Verify that the same values are returned from each run,
and update the bug with your results.
Once you verify that the updated prio checker no longer
generates change events, start up multipathd again and
verify that this continues to be true. Thanks!
** Attachment added: "updated netapp prio checker"
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2174493/+files/mpath_prio_netapp
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
I can understand that. Attached is a debdiff of the updated package.
I must stress that may not be the final form of the solution. I'm
chatting with Douglas Gilbert (sg io maintainer) now about this.
So far it appears that using an SD device as the target for sg io
can have unintended side effects, like the ones we've been observing.
If the corresponding SG device was used to begin with, this problem
would have never occurred.
See for yourself, mpath_prio_netapp /dev/sg0
In the meanwhile, this patch adjusts the open flag s.t. the "side effect"
no longer occurs. What's annoying is the upstream multipath (0.4.9) is
still using SD devices for this work but they did change all of their
open flags to O_RDONLY. Which leads me to believe they worked around
this side effect too, perhaps unaware of the fact that they were causing
these events to begin with by using sd devices by default.
You can simply build the mpath_prio_netapp straight from
path_priority/pp_netapp/ assuming build deps are met.
** Patch added: "preliminary patch, awaiting upstream feedback"
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2174791/+files/adjust-prioritizer-open-flags-to-avoid-sg-io-side-effects.debdiff
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
** Changed in: multipath-tools (Ubuntu)
Status: Confirmed => In Progress
** Changed in: udev (Ubuntu)
Status: New => Invalid
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
After chatting with Douglas Gilbert on this I know have a better
understanding of the problem. This very issue has been raised by
him before on linux-scsi and it never saw final resolution.
http://kerneltrap.org/mailarchive/linux-scsi/2010/2/15/6778453
When the file descriptor to the SD device is closed, the change
event occurs, presumably because it was opened with O_RDRW
to begin with, though nothing actually changed.
By changing that flag to O_RDONLY, no udev events are generated
when the fd closes. It doesn't address the root cause but it's
sufficient to get us unjammed safely while we continue to work
on a better solution. That solution may be adjusting the multipath
priority checkers to use the corresponding sg devices when
presented with an sd device. Either way, there's more work to do.
Since all the priority checkers amount to different degrees of
scsi inquiry, and vendor specific san interrogation commands. I
believe we're safe with the O_RDONLY approach. All data
direction flags for sg io in this case are SG_DXFER_FROM_DEV
or SG_DXFER_NONE.
** Patch added: "multipath-tools-eliminate-udev-change-events-lp644489.debdiff"
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+attachment/2177364/+files/multipath-tools-eliminate-udev-change-events-lp644489.debdiff
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
** Changed in: multipath-tools (Ubuntu)
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
also affects udev should be removed. ** Changed in: udev (Ubuntu Lucid) Status: New => Invalid ** Changed in: udev (Ubuntu Maverick) Status: New => Invalid ** Changed in: udev (Ubuntu Natty) Status: New => Invalid ** Package changed: udev (Ubuntu) => ubuntu ** Description changed: + = SRU Justification = + + == Impact == + Multipath-tools is inadvertedly generating UDEV CHANGE events for the SD + block devices under it's control. These change events feedback into the udev + rules, increasing cpu utlilzation, and ruining the multipath aliasing feature, + which allows one to rename a multipath path from a series of letters and + numbers, to a human readable label. It gives users the impression that + the SAN is unstable. + + == Solution == + Change the open() flags in the priority checkers to read only from read write, this + stops sd from generating a change event after the file descriptor has been + closed. + + Patch: https://bugs.launchpad.net/ubuntu/+source/multipath- + tools/+bug/644489/+attachment/2177364/+files/multipath-tools-eliminate- + udev-change-events-lp644489.debdiff + + == Reproduction == + + Is easy, and doesn't even require a SAN. Since we're dealing with simple SCSI + inquiry cmds any block device will do. Simply install multipath-tools and + execute one of the priority checkers like so: + + /sbin/mpath_prio_emc /dev/sda + + also, have a window open monitoring udev, udevadm monitor, ensure + no change events to that block device are occurring before hand. + + TEST CASE: + root@kickseed:~# udevadm monitor & + [1] 16950 + root@kickseed:~# monitor will print the received events for: + UDEV - the event which udev sends out after rule processing + KERNEL - the kernel uevent + root@kickseed:~# + root@kickseed:~# /sbin/mpath_prio_emc /dev/sda + query command indicates error0 + root@kickseed:~# KERNEL[1308688009.806317] change /devices/pci:00/:00:07.0/:04:00.0/host0/port-0:0/expander-0:0/port-0:0:1/end_device-0:0:1/target0:0:0/0:0:0:0/block/sda (block) + UDEV [1308688009.823569] change /devices/pci:00/:00:07.0/:04:00.0/host0/port-0:0/expander-0:0/port-0:0:1/end_device-0:0:1/target0:0:0/0:0:0:0/block/sda (block) + + root@kickseed:~# + root@kickseed:~# + root@kickseed:~# /sbin/mpath_prio_alua /dev/sda + 130 + + mpath_prio_alua doesn't generate any change events since it's open + flags do not include O_RDRW to begin with. + + == regression potential == + None, it's broken to begin with. + -- + Binary package hint: udev udevd constantly changes LUN device node symlinks (devices/LUNs, not the partition nodes) in /dev/disk/by-id. udevd uses ~15% of CPU and system time is using ~50-60%. For example: [j...@syslog01.roch.ny:pts/0 /dev/disk/by-id> ls -l wwn-0x60a98000486e5339576f596675735354 wwn-0x60a98000486e5339576f596675744c36 scsi-360a98000486e5339576f596675735354 scsi-360a98000486e5339576f596675744c36; sleep 1; echo '=='; ls -l wwn-0x60a98000486e5339576f596675735354 wwn-0x60a98000486e5339576f596675744c36 scsi-360a98000486e5339576f596675735354 scsi-360a98000486e5339576f596675744c36 lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675735354 -> ../../sde lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675744c36 -> ../../sdf lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675735354 -> ../../sde lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675744c36 -> ../../sdf == lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675735354 -> ../../sdg lrwxrwxrwx 1 root root 9 2010-09-21 16:12 scsi-360a98000486e5339576f596675744c36 -> ../../sdh lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675735354 -> ../../sdg lrwxrwxrwx 1 root root 9 2010-09-21 16:12 wwn-0x60a98000486e5339576f596675744c36 -> ../../sdh All other device nodes stay the same, such as the device nodes for the partitions: [j...@syslog01.roch.ny:pts/0 /dev/disk/by-id> ls -l scsi-360a98000486e5339576f596675735354-part1; sleep 1; echo '=='; ls -l scsi-360a98000486e5339576f596675735354-part1 lrwxrwxrwx 1 root root 10 2010-09-21 15:47 scsi-360a98000486e5339576f596675735354-part1 -> ../../sdg1 == lrwxrwxrwx 1 root root 10 2010-09-21 15:47 scsi-360a98000486e5339576f596675735354-part1 -> ../../sdg1 - - I'm not entirely sure whether this is udev's problem or something related to multipathing. Our most recent experience with multipathing is the last LTS release (hardy), which doesn't exhibit this behavior given similar configurations. - + I'm not entirely sure whether this is udev's problem or something + related to multipathing. Our most recent experience with multipathing is + the last LTS release (hardy), which doesn't exhibit this behavior given + similar configurations. [j...@syslog01.roch.ny:pts/0 ~> sudo multipath -ll - root
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
** Changed in: ubuntu
Status: Invalid => Confirmed
** Changed in: Ubuntu Lucid
Status: Invalid => Confirmed
** Changed in: Ubuntu Maverick
Status: Invalid => Confirmed
** Changed in: Ubuntu Natty
Status: Invalid => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 571093] Re: [SRU] multipath + libvirtd eats away more memory over time
@All, The stream of udev change events you've observed has been fixed in multipath itself by this update. https://launchpad.net/~peter-petrakis/+archive/storage It's been proposed for SRU on behalf of (LP: #644489) Please test and verify that the change events cease. So we can reconcile the multipath-tools status of this bug, Thanks! ** Changed in: multipath-tools (Ubuntu) Status: Triaged => Incomplete ** Changed in: multipath-tools (Ubuntu Lucid) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/571093 Title: [SRU] multipath + libvirtd eats away more memory over time To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/571093/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 716659] Re: [EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid
@Joe What's the status of this? Is the issue resolved? Thanks. ** Changed in: multipath-tools (Ubuntu) Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/716659 Title: [EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/716659/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 716659] Re: [EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid
Moving visibility to private since this is an resolved customer support issue. We can't fix it if they're not willing to work with us. That probably means either remote access or on site troubleshooting. Peter ** Visibility changed to: Private -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/716659 Title: [EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/716659/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
@ Martin
fix verified, no udev change events generated.
root@kickseed:~# udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
^Z
[1]+ Stopped udevadm monitor
root@kickseed:~# bg
[1]+ udevadm monitor &
root@kickseed:~# apt-cache policy multipath-tools
multipath-tools:
Installed: 0.4.8-14ubuntu10.1
Candidate: 0.4.8-14ubuntu10.1
Version table:
*** 0.4.8-14ubuntu10.1 0
500 http://us.archive.ubuntu.com/ubuntu/ natty-proposed/main Packages
100 /var/lib/dpkg/status
0.4.8-14ubuntu10 0
500 http://10.229.0.1/enablement/lucid-mpio-ubuntu/ lucid/main Packages
0.4.8-14ubuntu4 0
500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages
root@kickseed:~#
root@kickseed:~# /sbin/mpath_prio_netapp /dev/sda
SCSI error
- SCSI CDB: 0xc0 0x00 0x01 0x0a 0x98 0x0a 0x41 0x80 0x00 0x00
- masked_status=0x01, host_status=0x00, driver_status=0x08
- SCSI sense data: 0x70 0x00 0x05 0x00 0x00 0x00 0x00 0x0c 0x00 0x00 0x00 0x00
0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00
SCSI error
- SCSI CDB: 0x12 0x01 0xc1 0x00 0x80 0x00
- masked_status=0x01, host_status=0x00, driver_status=0x08
- SCSI sense data: 0x70 0x00 0x05 0x00 0x00 0x00 0x00 0x0c 0x00 0x00 0x00 0x00
0x24 0x00 0x00 0x00 0x00 0x00 0x00 0x00
4
root@kickseed:~#
root@kickseed:~#
root@kickseed:~# /sbin/mpath_prio_netapp /dev/sdb
SCSI error
- SCSI CDB: 0xc0 0x00 0x01 0x0a 0x98 0x0a 0x41 0x80 0x00 0x00
- masked_status=0x01, host_status=0x00, driver_status=0x08
- SCSI sense data: 0x70 0x00 0x05 0x00 0x00 0x00 0x00 0x0c 0x00 0x00 0x00 0x00
0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00
SCSI error
- SCSI CDB: 0x12 0x01 0xc1 0x00 0x80 0x00
- masked_status=0x01, host_status=0x00, driver_status=0x08
- SCSI sense data: 0x70 0x00 0x05 0x00 0x00 0x00 0x00 0x0c 0x00 0x00 0x00 0x00
0x24 0x00 0x00 0x00 0x00 0x00 0x00 0x00
4
root@kickseed:~# /sbin/mpath_prio_emc /dev/sdb
query command indicates error0
root@kickseed:~#
root@kickseed:~#
root@kickseed:~# /sbin/mpath_prio_emc /dev/sda
query command indicates error0
root@kickseed:~#
root@kickseed:~#
root@kickseed:~# jobs
[1]+ Running udevadm monitor &
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
@Clint
Maverick verification: no udev change events observed
root@kickseed:~# udevadm monitor &
[1] 1999
root@kickseed:~# monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
root@kickseed:~# for i in /sbin/mpath_prio_*
> do
> echo "testing pathchecker $i"
> $i /dev/sda 2>/dev/null
> done
testing pathchecker /sbin/mpath_prio_alua
50
testing pathchecker /sbin/mpath_prio_balance_units
1
testing pathchecker /sbin/mpath_prio_emc
0
testing pathchecker /sbin/mpath_prio_hds_modular
0
testing pathchecker /sbin/mpath_prio_hp_sw
4
testing pathchecker /sbin/mpath_prio_netapp
4
testing pathchecker /sbin/mpath_prio_random
7
testing pathchecker /sbin/mpath_prio_rdac
0
root@kickseed:~# uname -a
Linux kickseed 2.6.35-30-generic #54-Ubuntu SMP Tue Jun 7 18:41:54 UTC 2011
x86_64 GNU/Linux
root@kickseed:~# apt-cache policy multipath-tools
multipath-tools:
Installed: 0.4.8-14ubuntu4.10.10.1
Candidate: 0.4.8-14ubuntu4.10.10.1
Version table:
*** 0.4.8-14ubuntu4.10.10.1 0
500 http://us.archive.ubuntu.com/ubuntu/ maverick-proposed/main amd64
Packages
100 /var/lib/dpkg/status
0.4.8-14ubuntu4 0
500 http://us.archive.ubuntu.com/ubuntu/ maverick/main amd64 Packages
root@kickseed:~# mount
/dev/mapper/2226500015588d809-part1 on / type ext4 (rw,errors=remount-ro)
...
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
@Clint
lucid verification: no udev change events observed
NOTE: this issue depends on lp #686832
root@kickseed:~# lsb_release -r
Release:10.04
root@kickseed:~# udevadm monitor &
[1] 2043
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
root@kickseed:~# for i in /sbin/mpath_prio_*
> do
> echo "testing pathchecker $i"
> $i /dev/sda 2>/dev/null
> done
testing pathchecker /sbin/mpath_prio_alua
50
testing pathchecker /sbin/mpath_prio_balance_units
1
testing pathchecker /sbin/mpath_prio_emc
0
testing pathchecker /sbin/mpath_prio_hds_modular
0
testing pathchecker /sbin/mpath_prio_hp_sw
4
testing pathchecker /sbin/mpath_prio_netapp
4
testing pathchecker /sbin/mpath_prio_random
1
testing pathchecker /sbin/mpath_prio_rdac
0
root@kickseed:~# uname -a
Linux kickseed 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 21:52:38 UTC 2011
x86_64 GNU/Linux
root@kickseed:~# apt-cache policy multipath-tools
multipath-tools:
Installed: 0.4.8-14ubuntu4.10.04.1
Candidate: 0.4.8-14ubuntu4.10.04.1
Version table:
*** 0.4.8-14ubuntu4.10.04.1 0
500 http://us.archive.ubuntu.com/ubuntu/ lucid-proposed/main Packages
100 /var/lib/dpkg/status
0.4.8-14ubuntu4 0
500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages
root@kickseed:~# mount
/dev/mapper/222515568f586-part1 on / type ext4 (rw,errors=remount-ro)
...
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 644489] Re: constantly changes /dev/disk/by-id/{scsi, wwn}-* LUN symlinks with multipathing
On 06/28/2011 09:30 PM, Serge Hallyn wrote:
> Quoting Peter Petrakis (peter.petra...@canonical.com):
>> @Clint
>>
>> lucid verification: no udev change events observed
>> NOTE: this issue depends on lp #686832
>
> Not strictly though, right? Your setup needed that fix to boot,
> but that is unrelated to this bugfix iiuc?
That's exactly correct. Just ensuring the dependency, that before
multipath-tools lands in lucid-updates, lp #686832 must
be integrated. I don't want some random user taking this
deb and expecting their system will improve as a whole; they'll
be disappointed when they are unable to boot. Other than that,
consider this fix verified.
Peter
>
> -serge
>
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/644489
Title:
constantly changes /dev/disk/by-id/{scsi,wwn}-* LUN symlinks with
multipathing
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/644489/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 803554] [NEW] upgrading to multipath 0.4.9 presents incompatible config file changes
Public bug reported: The priority checkers have been converted from individual binaries that callout to dynamically loaded libraries (dlopen). As a result, should a circa 0.4.8 config file remain in place, the priority checker becomes undefined, as that standalone path checker no longer exists. This will effect users who upgrade to Oneiric and users who upgrade from lucid to the next LTS. One example of the format change is, 0.4.8: prio_callout"/sbin/mpath_prio_alua /dev/%n" 0.4.9: prioalua There may be other config file changes, more exploration is needed. Here's what the preconfigured, 0.4.9 priority checkers look like. # echo 'show config' | multipathd -k | grep 'prio '| sort -u prio alua prio const prio emc prio hds prio hp_sw prio netapp prio rdac In 0.4.8: # echo 'show config' | multipathd -k | grep 'prio_callout'| sort -u prio_callout /sbin/mpath_prio_alua /dev/%n prio_callout /sbin/mpath_prio_alua %n prio_callout /sbin/mpath_prio_emc /dev/%n prio_callout /sbin/mpath_prio_hds_modular /dev/%n prio_callout /sbin/mpath_prio_hp_sw /dev/%n prio_callout /sbin/mpath_prio_netapp /dev/%n prio_callout /sbin/mpath_prio_rdac /dev/%n ** Affects: multipath-tools (Ubuntu) Importance: Undecided Status: Confirmed ** Description changed: - The priority checkers have been converted from individual binaries that callout - to dynamically loaded libraries (dlopen). As a result, should a circa 0.4.8 config file remain - in place, the priority checker becomes undefined, as that standalone path checker - no longer exists. This will effect users who upgrade to Oneiric and users who upgrade - from lucid to the next LTS. + The priority checkers have been converted from individual binaries that + callout to dynamically loaded libraries (dlopen). As a result, should a + circa 0.4.8 config file remain in place, the priority checker becomes + undefined, as that standalone path checker no longer exists. This will + effect users who upgrade to Oneiric and users who upgrade from lucid to + the next LTS. One example of the format change is, 0.4.8: prio_callout"/sbin/mpath_prio_alua /dev/%n" 0.4.9: prioalua There may be other config file changes, more exploration is needed. Here's what the preconfigured, 0.4.9 priority checkers look like. # echo 'show config' | multipathd -k | grep 'prio '| sort -u - prio alua - prio const - prio emc - prio hds - prio hp_sw - prio netapp - prio rdac + prio alua + prio const + prio emc + prio hds + prio hp_sw + prio netapp + prio rdac In 0.4.8: # echo 'show config' | multipathd -k | grep 'prio_callout'| sort -u - prio_callout /sbin/mpath_prio_alua /dev/%n - prio_callout /sbin/mpath_prio_alua %n - prio_callout /sbin/mpath_prio_emc /dev/%n - prio_callout /sbin/mpath_prio_hds_modular /dev/%n - prio_callout /sbin/mpath_prio_hp_sw /dev/%n - prio_callout /sbin/mpath_prio_netapp /dev/%n - prio_callout /sbin/mpath_prio_rdac /dev/%n + prio_callout /sbin/mpath_prio_alua /dev/%n + prio_callout /sbin/mpath_prio_alua %n + prio_callout /sbin/mpath_prio_emc /dev/%n + prio_callout /sbin/mpath_prio_hds_modular /dev/%n + prio_callout /sbin/mpath_prio_hp_sw /dev/%n + prio_callout /sbin/mpath_prio_netapp /dev/%n + prio_callout /sbin/mpath_prio_rdac /dev/%n ** Changed in: multipath-tools (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/803554 Title: upgrading to multipath 0.4.9 presents incompatible config file changes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/803554/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 916200] Re: missing kernel module ipmi_si module
We need a full apport collect to see the fall out from loading the module. apport-collect -p linux 916200 Thanks. ** Changed in: ipmitool (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ipmitool in Ubuntu. https://bugs.launchpad.net/bugs/916200 Title: missing kernel module ipmi_si module To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ipmitool/+bug/916200/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 916200] Re: missing kernel module ipmi_si module
@Asif Get the apport data any way you can. I merely offered a popular manner to accomplish it. Also, please don't change bug states without justification. The point of Incomplete is part of the bug triage process where a developer has posed a question etc and is waiting for that request to be satisfied. It doesn't lower the priority of your issue, in fact it draws more attention to it. Peter ** Changed in: ipmitool (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ipmitool in Ubuntu. https://bugs.launchpad.net/bugs/916200 Title: missing kernel module ipmi_si module To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ipmitool/+bug/916200/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 916200] Re: missing kernel module ipmi_si module
** Changed in: ipmitool (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ipmitool in Ubuntu. https://bugs.launchpad.net/bugs/916200 Title: missing kernel module ipmi_si module To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ipmitool/+bug/916200/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 956128] [NEW] find_multipaths feature missing from upstream
Public bug reported: There's an interesting feature called find_multipaths that was first mentioned here on dm-devel in July 2011. https://www.redhat.com/archives/dm-devel/2011-July/msg00102.html It's now the corner stone of dm multipath configuration in the RHEL 6 DM Multipath configuration guide https://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/config_file_blacklist.html What it basically allows is autoprobing of devices to safely enable multpath on luns that satisfy the proper criteria. It has the power to reduce a multipath.conf configuration to a single line. Alas it doesn't exist in the upstream sources, only as a patch in RH's multipath package 0021-RHBZ-558636-add-find-multipaths.patch This bug exists to track the status of this feature and it's ultimate integration into upstream or into our multipath-tools package. ** Affects: multipath-tools (Ubuntu) Importance: Undecided Status: New ** Tags: precise -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/956128 Title: find_multipaths feature missing from upstream To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/956128/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 956128] Re: find_multipaths feature missing from upstream
** Attachment added: "0021-RHBZ-558636-add-find-multipaths.patch" https://bugs.launchpad.net/bugs/956128/+attachment/2876175/+files/0021-RHBZ-558636-add-find-multipaths.patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/956128 Title: find_multipaths feature missing from upstream To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/956128/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985741] Re: multipath can't show device on which is setup lvm
I don't think you need to upgrade at all, your multipathd is sufficiently modern and not much different from the version in 12.04. Interesting, I see that you're using user_friendly_names, though since you're also using LVM, it kind of makes that feature redundant as the LVM is now your user friendly name, and having that multipath feature adds another level of complexity when UDEV is creating devices. Please set "user_friendly_names no" in /etc/multipath.conf, and update your initrd "update-initramfs -k all -u". The last part is important as the multipath.conf is copied to the initrd and used as the basis for any blacklists or additional UDEV juggling. What might have happened is you didn't update the initrd since you created the initial multipath config. Since the default is no, the ramdisk creates the normal long style WWID names, then it mounts root, multipathd starts which finds user friendly names set and then begins destroying the original WWID names in exchange for the mpathx versions. If LVM has claimed the MP device by then, destruction of the original name (and dmtable entry) will be blocked by reference counting. In general, it's imperative that you update the initramfs whenever lvm.conf or multipath.conf is changed, especially when you're using them together. https://help.ubuntu.com/12.04/serverguide/multipath-devices.html #multipath-devices-in-logical-volumes Also, please adjust your local character encoding to a western font. Some of your text files look like gibberish e.g. https://launchpadlibrarian.net/103100334/test_before_after_lvm_reboot.txt a¡±¡±a¡±€ (8:144) mpath9 (252:7) a¡±?a¡±€ (65:80) a¡±¡±a¡±€ (8:160) mpath15 (252:12) a¡±?a¡±€ (8:224) a¡±¡±a¡±€ (8:48) mpath5-part1 (252:10) a¡±¡±a¡±€mpath5 (252:2) a¡±?a¡±€ (65:16) a¡±¡±a¡±€ (8:96) mpath14-part1 (252:9) Thank you. -- Peter ** Changed in: multipath-tools (Ubuntu) Status: Confirmed => Incomplete ** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Peter Petrakis (peter-petrakis) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/985741 Title: multipath can't show device on which is setup lvm To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/985741/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 985741] Re: multipath can't show device on which is setup lvm
@Vincent,
I'm glad we could get you sorted. New for 12.04 is the DM-Multipath
documentation which
really focuses around the upstream version multipath-0.4.9. You may find other
useful
bits of information there. Closing out this issue as "Invalid" since it is not
a defect.
https://help.ubuntu.com/12.04/serverguide/dm-multipath-chapter.html
Take care.
-- Peter
** Changed in: multipath-tools (Ubuntu)
Status: Incomplete => Invalid
** Changed in: multipath-tools (Ubuntu)
Assignee: Peter Petrakis (peter-petrakis) => (unassigned)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/985741
Title:
multipath can't show device on which is setup lvm
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/985741/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1032550] Re: [multipath] failed to get sysfs information
** Changed in: multipath-tools (Ubuntu) Assignee: Peter Petrakis (peter-petrakis) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/1032550 Title: [multipath] failed to get sysfs information To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1032550/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 690387] [NEW] udev block naming breaks failover and sd kref release cycle
Public bug reported: Binary package hint: multipath-tools This was exposed on the Intel IMS SAN which is an ODM'd Promise Vtrak variant on 10.04 server. The SAN has Active/Standby capabilities and is configured for failover. It probably affects other SANs too. Setup: multipath'd SAN consisting of SD block devices. Symptoms: On failover, multipath isn't gettng the right signals to tear down the defunct path. This was traced down to the fact that the path UDEV was presenting to multipath was different from what it was expecting. It simply dropped the request to gracefully remove the device, and instead responded to the SCSI mid-layer SD IO state change, SDEV_CANCEL/DEL which puts the device offline. Problem is device mapper still has an handle on the SD device, as can be seen from /sys/block/dm-x/slaves, and as a result, scsi_target_destroy is never called. The outward symptom of this is the SD suffix is not recycled because of course the previous reference never dropped. Solution: A fix was developed independently of upstream by Serge Hallyn, later it was found that it was fixed upstream, in 2008. The patch is: commit 7fa7affc3d23dd9dc906804d22a61144bca9f9b9 Author: Benjamin Marzinski Date: Thu Dec 11 16:03:28 2008 -0600 Fix for uevent devpath handling This is necessary to make uevents work on fedora, since devpath appears as something like: '/devices/pci:00/:00:0a.0/:06:00.0/host11/rport-11:0-1/target11:0:1/11:0:1:0/block/s It simply strips off the everything up to the /block. Signed-off-by: Benjamin Marzinski It integrates simply and can be found in PPAs here: ppa:peter-petrakis/storage ** Affects: multipath-tools (Ubuntu) Importance: High Assignee: Serge Hallyn (serge-hallyn) Status: Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in ubuntu. https://bugs.launchpad.net/bugs/690387 Title: udev block naming breaks failover and sd kref release cycle -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 690387] Re: udev block naming breaks failover and sd kref release cycle
** Patch added: "0001-Fix-for-uevent-devpath-handling.patch" https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/690387/+attachment/1766245/+files/0001-Fix-for-uevent-devpath-handling.patch -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in ubuntu. https://bugs.launchpad.net/bugs/690387 Title: udev block naming breaks failover and sd kref release cycle -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 690387] Re: udev block naming breaks failover and sd kref release cycle
** Description changed: Binary package hint: multipath-tools - This was exposed on the Intel IMS SAN which is an ODM'd Promise Vtrak variant - on 10.04 server. The SAN has Active/Standby capabilities and is configured for failover. - It probably affects other SANs too. + This was exposed on the Intel IMS SAN which is an ODM'd Promise Vtrak + variant on 10.04 server. The SAN has Active/Standby capabilities and + is configured for failover. It probably affects other SANs too. Setup: multipath'd SAN consisting of SD block devices. Symptoms: - On failover, multipath isn't gettng the right signals to tear down the defunct - path. This was traced down to the fact that the path UDEV was presenting to - multipath was different from what it was expecting. It simply dropped the request - to gracefully remove the device, and instead responded to the SCSI mid-layer - SD IO state change, SDEV_CANCEL/DEL which puts the device offline. + On failover, multipath isn't gettng the right signals to tear down + the defunct path. This was traced down to the fact that the path UDEV + was presenting to multipath was different from what it was expecting. + It simply dropped the request to gracefully remove the device, and + instead responded to the SCSI mid-layer SD IO state change, + SDEV_CANCEL/DEL which puts the device offline. - Problem is device mapper still has an handle on the SD device, as can be seen - from /sys/block/dm-x/slaves, and as a result, scsi_target_destroy is never called. - The outward symptom of this is the SD suffix is not recycled because of course - the previous reference never dropped. + Problem is device mapper still has an handle on the SD device, as + can be seen from /sys/block/dm-x/slaves, and as a result, + scsi_target_destroy is never called. The outward symptom of this + is the SD suffix is not recycled because of course the previous + reference never dropped. Solution: - A fix was developed independently of upstream by Serge Hallyn, later it - was found that it was fixed upstream, in 2007. The patch is. + A fix was developed independently of upstream by Serge Hallyn, + later it was found that it was fixed upstream, in 2008. + The patch is: commit 7fa7affc3d23dd9dc906804d22a61144bca9f9b9 Author: Benjamin Marzinski Date: Thu Dec 11 16:03:28 2008 -0600 - Fix for uevent devpath handling + Fix for uevent devpath handling - This is necessary to make uevents work on fedora, since devpath appears as - something like: - '/devices/pci:00/:00:0a.0/:06:00.0/host11/rport-11:0-1/target11:0:1/11:0:1:0/block/s + This is necessary to make uevents work on fedora, since devpath appears as + something like: + '/devices/pci:00/:00:0a.0/:06:00.0/host11/rport-11:0-1/target11:0:1/11:0:1:0/block/s - It simply strips off the everything up to the /block. + It simply strips off the everything up to the /block. - Signed-off-by: Benjamin Marzinski + Signed-off-by: Benjamin Marzinski It integrates simply and can be found in PPAs here: ppa:peter-petrakis/storage -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in ubuntu. https://bugs.launchpad.net/bugs/690387 Title: udev block naming breaks failover and sd kref release cycle -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 716659] Re: Root filesystem goes "Read only" after installing multipath-tools on Lucid
A few things, ask the customer to run the command 'multipath -v4'
as root after the package install failure so we can see what's going on.
There's at least one bug in the multipath.conf
1) scsi_id path is wrong
2) the blacklist regex is suspect
1)
Replace all calls to scsi id with:
"/lib/udev/scsi_id -g -u -d /dev/%n"
2) This regex assumes the sd names are ordered, this is never true.
Why are these sd devices being blacklisted to begin with?
blacklist {
devnode "^sda[1-7]"
devnode "^sd[f-z]"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
SD driver pre-allocates up to 15 partitions per whole block device,
the first regex only filters out the 1/2 of the possible partitions of
the first probed device.
If I'm to believe this blacklist then we're looking to serve
sda8-15, sdb, c,d,e and that's it.
I don't think this blacklist is viable, try commenting out the first two
devnodes.
Please try each change independently and as a whole, and report.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in ubuntu.
https://bugs.launchpad.net/bugs/716659
Title:
Root filesystem goes "Read only" after installing multipath-tools on
Lucid
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 690387] Re: udev block naming breaks failover and sd kref release cycle
@Serge Yes, fix-released for oneiric I believe natty is covered. The confusion I think is the changelog references this fix from a private bug: multipath-tools (0.4.8-14ubuntu7) natty; urgency=low * Add patch to fix the expected pathname from multipath uevents (LP: #660597) Where this bug, lp #690387 is the public instance of the fix. Though in lucid, I don't see either bug number mentioned in the changelog. multipath-tools (0.4.8-14ubuntu4.10.04.1) lucid-proposed; urgency=high * Eliminate UDEV CHANGE events generated by mpath priority checkers. Due to quirk in how SG IO is handled by SD devices (LP: #644489). -- Peter M. Petrakis Tue, 21 Jun 2011 14:13:12 -0400 multipath-tools (0.4.8-14ubuntu4) lucid; urgency=low * debian/control: Move libreadline5-dev build dependency to libreadline-dev. (Already done in Debian). -- Martin Pitt Mon, 08 Mar 2010 14:50:04 +0100 multipath-tools (0.4.8-14ubuntu3) lucid; urgency=low * Support failback for Intel Modular Server (LP: #520309). -- Colin Watson Fri, 12 Feb 2010 12:21:42 + root@kickseed:~# apt-cache policy multipath-tools multipath-tools: Installed: 0.4.8-14ubuntu4.10.04.1 Candidate: 0.4.8-14ubuntu4.10.04.1 Version table: 0.4.8-14ubuntu10.1 0 800 http://us.archive.ubuntu.com/ubuntu/ natty-updates/main Packages 0.4.8-14ubuntu10 0 800 http://us.archive.ubuntu.com/ubuntu/ natty/main Packages *** 0.4.8-14ubuntu4.10.04.1 0 900 http://us.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages 100 /var/lib/dpkg/status 0.4.8-14ubuntu4 0 900 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages Do we have this pending SRU for lucid somewhere and lost track of it? http://people.canonical.com/~ubuntu-archive/pending-sru.html I don't see anything pending for multipath. Wasn't there a mass changeset planned for multipath lucid SRU that included this fix? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/690387 Title: udev block naming breaks failover and sd kref release cycle To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/690387/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 803554] Re: upgrading to multipath 0.4.9 presents incompatible config file changes
Experimenting shows that we can have the older config values side by side with the new ones. e.g. ... prio_callout /sbin/mpath_prio_alua /dev/%n prio alua ... multipath doesn't have a rigid config file format, it seems to slurp symbols and then seek the key/value pair out on demand. In doing this, we can safely downgrade as well. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/803554 Title: upgrading to multipath 0.4.9 presents incompatible config file changes To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/803554/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 690387] Re: udev block naming breaks failover and sd kref release cycle
@Clint lucid-proposed looks good. $ apt-cache showpkg multipath-tools Package: multipath-tools Versions: 0.4.8-14ubuntu4.10.04.2 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_lucid-proposed_main_binary-amd64_Packages) (/var/lib/dpkg/status) # multipath -ll 222fb00015553d0d9dm-3 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:1:1 sdf 8:80 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:1 sdb 8:16 [active][ready] 2225f00015562a505dm-1 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:1:3 sdh 8:112 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:3 sdd 8:48 [active][ready] 222515568f586dm-2 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:1:0 sde 8:64 [active][ready] 222d8000155636433dm-0 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:2 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:1:2 sdg 8:96 [active][ready] # Reset a controller # multipath -ll 222fb00015553d0d9dm-3 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=50][active] \_ 0:0:0:1 sdb 8:16 [active][ready] 2225f00015562a505dm-1 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=50][active] \_ 0:0:0:3 sdd 8:48 [active][ready] 222515568f586dm-2 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:0 sda 8:0 [active][ready] 222d8000155636433dm-0 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:2 sdc 8:32 [active][ready] # It's back # multipath -ll 222fb00015553d0d9dm-3 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:2:1 sdf 8:80 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:1 sdb 8:16 [active][ready] 2225f00015562a505dm-1 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:2:3 sdh 8:112 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:3 sdd 8:48 [active][ready] 222515568f586dm-2 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:2:0 sde 8:64 [active][ready] 222d8000155636433dm-0 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:2 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:2:2 sdg 8:96 [active][ready] # SD names were recycled correctly - FIXED -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/690387 Title: udev block naming breaks failover and sd kref release cycle To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/690387/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 690387] Re: udev block naming breaks failover and sd kref release cycle
@Clint maverick-proposed looks good. # apt-cache showpkg multipath-tools Package: multipath-tools Versions: 0.4.8-14ubuntu4.10.10.2 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_maverick-proposed_main_binary-amd64_Packages) (/var/lib/dpkg/status) # multipath -ll 222fb00015553d0d9dm-0 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:1:1 sdf 8:80 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:1 sdb 8:16 [active][ready] 2225f00015562a505dm-1 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:1:3 sdh 8:112 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:3 sdd 8:48 [active][ready] 222515568f586dm-3 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:1:0 sde 8:64 [active][ready] 222d8000155636433dm-2 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:2 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:1:2 sdg 8:96 [active][ready] # Reset a controller # multipath -ll 222fb00015553d0d9dm-0 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=50][active] \_ 0:0:0:1 sdb 8:16 [active][ready] 2225f00015562a505dm-1 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=50][active] \_ 0:0:0:3 sdd 8:48 [active][ready] 222515568f586dm-3 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][enabled] \_ 0:0:0:0 sda 8:0 [active][ready] 222d8000155636433dm-2 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:2 sdc 8:32 [active][ready] # It's back # multipath -ll 222fb00015553d0d9dm-0 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:2:1 sdf 8:80 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:1 sdb 8:16 [active][ready] 2225f00015562a505dm-1 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:2:3 sdh 8:112 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:0:3 sdd 8:48 [active][ready] 222515568f586dm-3 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:0 sda 8:0 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:2:0 sde 8:64 [active][ready] 222d8000155636433dm-2 Intel ,Multi-Flex [size=10G][features=1 queue_if_no_path][hwhandler=1 alua] \_ round-robin 0 [prio=130][active] \_ 0:0:0:2 sdc 8:32 [active][ready] \_ round-robin 0 [prio=1][enabled] \_ 0:0:2:2 sdg 8:96 [active][ready] # SD names were recycled correctly - FIXED -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/690387 Title: udev block naming breaks failover and sd kref release cycle To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/690387/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 824790] Re: IBM DS3400 Will Not Bring Up Second Path
@Joe Please attach the latest multipath.conf and lvm.conf responsible for this stream of messages. If you could move over that logs and other stuff from salesforce into this bug that would be ideal. Thanks. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/824790 Title: IBM DS3400 Will Not Bring Up Second Path To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/824790/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 824790] Re: IBM DS3400 Will Not Bring Up Second Path
So the paths not showing up with a config file error, a few well
placed regexp and a proper blacklist to filter out the internal disks
solved that. Also added scsi_dh_rdac to initial ramdisk. Now the paths
are up, and stay up. However when we tried testing failover using
a scsi "delete" of the primary member. The secondary never became
active.
We asked the customer to check with any additional SAN side
config, like automatic lun transfer, is necessary. Depending on how
that turns it we may actually have a real failover bug.
Working multipath config, taken almost verbatim from the IBM
documentation: Section 5-10.
ftp://ftp.software.ibm.com/systems/support/system_x_pdf/94y8402.pdf
# Canonical Customer Support: 08/12/11
# Version 1.2
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
# Ignore internal storage from the LSI RAID
device {
vendor LSILOGIC
product .*
}
}
devices {
device {
vendor "IBM"
product ".*"
path_grouping_policy group_by_prio
path_checker rdac
getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"
prio_callout "/sbin/mpath_prio_rdac /dev/%n"
hardware_handler "1 rdac"
failback immediate
#rr_weightuniform
no_path_retry30
rr_min_io100
features "2 pg_init_retries 50"
}
}
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/824790
Title:
IBM DS3400 Will Not Bring Up Second Path
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/824790/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 824790] Re: IBM DS3400 Will Not Bring Up Second Path
** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Ubuntu Storage Development Team (ubuntu-storage-dev) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/824790 Title: IBM DS3400 Will Not Bring Up Second Path To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/824790/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 824790] Re: IBM DS3400 Will Not Bring Up Second Path
** Changed in: multipath-tools (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to multipath-tools in Ubuntu. https://bugs.launchpad.net/bugs/824790 Title: IBM DS3400 Will Not Bring Up Second Path To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/824790/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
