Can the bits that make those devices invalid and udev out of date call udev admin --settle to just block till things are upto date and hopefully the subseqent vg and pv scans quicker?
On Monday, March 16, 2015, John Griffith <john.griffi...@gmail.com> wrote: > Hey Everyone, > > Thought I'd reach out to the ML to see if somebody might have some insight > or suggestions to a problem I've been trying to solve. > > The short summary is: > > During a dsvm-full run in the gate there are times when /dev/sdX devices > on the system may be created/deleted. The trick here though is that on the > Cinder side with LVM we're doing a lot of things that rely on VGS and LVS > calls (including periodic tasks that are run). Part of the scan routine > unfortunately is for LVM to go through and open any block devices that is > sees on the system and read them to see if it's an LVM device. > > The problem here is that the timing in the gate tests when everything is > on a single node can result in udev not being quite up to date and the LVM > scan process attempts to open a device that is no longer valid. In this > case (which we hit a few times on every single gate test), LVM blocks on > the Open until the device times out and gives: > -1 ENXIO (No such device or address) > > The problem is this can take up to almost a full minute for the timeout, > so we have a few tests that take upwards of 150 seconds that actually > should complete in about 30 seconds. In addition this causes things like > concurrent lvcreate cmds to block as well. Note this is kind of > inefficient anyway (even if all of the devices are valid), so there is a > case to be made for not doing it if possible. > > Nothing fails typically in this scenario, things are just "slow". > > I thought this would be easy to fix a while back by adding a local > lvm.conf with a device filter. It turns out however that the device filter > only filters out items AFTER the vgs or lvs, it doesn't filter out the > opens. For that you need either: > 1. global_filter > 2. lvmetad service enabled > > The problem with '#1' is that the global_filter parameter is only honored > on a global level NOT just in a local lvm.conf like we have currently. To > use that though we would have to set things such that Cinder was the only > thing using LVM (not sure if that's doable or not). > > The problem with '#2' is that Trusty doesn't seem to have lvmetad in it's > lvm2 packages until 2.02.111 (which isn't introduced until Vivid). > > I'm wondering if anybody knows of a backport or another method to get > lvmetad capability in Ubuntu Trusty? > > OR > > What are some thoughts regarding the addition of a global_filter in > /etc/lvm.conf? We'd have to make a number of modifications to any services > in devstack that setup LVM to make sure their PV's are added to the > filter. This might not be a big deal because most everyone uses loopback > files so we could just to a loop regex and hit everything in one shot (I > think). But this means that anybody using LVM in devstack for something > else is going to need to understand what's going on and add their devices > to the filter. > > Ok... so not really the short version after all, but I'm wondering if > anybody has any ideas that maybe I'm missing here. I'll likely proceed > with the idea of a global filter later this week if I don't hear any strong > objections, or even better maybe somebody knows how to get lvmetad on > Trusty which I *think* would be ideal for a number of other reasons. > > Thanks, > John >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
