On Wed, Dec 2, 2020 at 4:57 PM Joy Li <joooy...@gmail.com> wrote:
>
> Thanks a lot Nir! Good to know that oVirt cannot guarantee the disk names so
> that I don't need to spend more time trying to enable such a feature.
>
> I can always reproduce the problem via my application, basically, the
> procedure is as following:
>
> 1. create a VM
Which guest OS? Can you share the guest disk image or ISO image used
to instal it?
> 2. add disks to the VM (e.g.: disk names: disk1, disk3)
> 3. check the disk mappings via `virsh domblklist `
Please shared the libvirt domain xml (virsh dumpxml vm-name)
> 4. add another disk (let's say, disk2, give a name alphabetically before some
> existing disks)
Add disk while the vm is running (hotplug)?
> 5. shutdown the VM via hypervisor and start it again (reboot won't work)
What do you mean by "reboot does not work?"
> 6. `virsh domblklist` again, then you might see the problem I mentioned before
Mapping is different compared with state before the reboot?
> There are no virtio devices inside /dev/disk/by-id/xxx of my guest VM.
Maybe you don't have systemd-udev installed?
The links in /dev/disk/... are created by udev during startup, and
when detecting a new disk.
> And I just noticed that the disks mapping information given by hypervisor
> (from VM configuration or virsh command) is different from the reality inside
> the VM. The disk name inside the VM was actually not changed.
>
> So now my issue is that given a disk name (/dev/vdb) of a VM, how can I get
> its wwid? Before I got it from the hypervisor, but now the hypervisor's
> information is not reliable, and since the disk is unformatted, I cannot use
> UUID.
You can use:
# udevadm info /dev/vda
P: /devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio4/block/vda
N: vda
L: 0
S: disk/by-path/pci-0000:06:00.0
S: disk/by-id/virtio-b97e68b2-87ea-45ca-9
S: disk/by-path/virtio-pci-0000:06:00.0
E: DEVPATH=/devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio4/block/vda
E: DEVNAME=/dev/vda
E: DEVTYPE=disk
E: MAJOR=252
E: MINOR=0
E: SUBSYSTEM=block
E: USEC_INITIALIZED=10518442
E: ID_SERIAL=b97e68b2-87ea-45ca-9
E: ID_PATH=pci-0000:06:00.0
E: ID_PATH_TAG=pci-0000_06_00_0
E: DEVLINKS=/dev/disk/by-path/pci-0000:06:00.0
/dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9
/dev/disk/by-path/virtio-pci-0000:06:00.0
E: TAGS=:systemd:
I tried to reproduce you issue 4.4.5 development build:
Starting vm with 2 direct LUN disks:
# virsh -r dumpxml disk-mapping
...
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='qcow2' cache='none'
error_policy='stop' io='native' discard='unmap'/>
<source
dev='/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5'
index='3'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore type='block' index='5'>
<format type='qcow2'/>
<source
dev='/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/ad891316-b80a-4ab5-895b-400108bd0ca1'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
</backingStore>
<target dev='sda' bus='scsi'/>
<serial>40018b33-2b11-4d10-82e4-604a5b135fb2</serial>
<boot order='1'/>
<alias name='ua-40018b33-2b11-4d10-82e4-604a5b135fb2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/3600140594af345ed76d42058f2b1a454' index='2'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>b97e68b2-87ea-45ca-94fb-277d5b30baa2</serial>
<alias name='ua-b97e68b2-87ea-45ca-94fb-277d5b30baa2'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5' index='1'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>d9a29187-f492-4a0d-aea2-7d5216c957d7</serial>
<alias name='ua-d9a29187-f492-4a0d-aea2-7d5216c957d7'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
</disk>
...
# virsh -r domblklist disk-mapping
Target Source
---------------------------------------------------------------------------------------------------------------------------------------------------------------
sdc -
sda
/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5
vda /dev/mapper/3600140594af345ed76d42058f2b1a454
vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5
In guest:
# ls -lh /dev/disk/by-id/virtio-*
lrwxrwxrwx. 1 root root 9 Jan 6 09:42
/dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda
lrwxrwxrwx. 1 root root 9 Jan 6 09:42
/dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb
NOTE: "d9a29187-f492-4a0d-a" are the first characters of the disk id:
"d9a29187-f492-4a0d-aea2-7d5216c957d7"
seen in oVirt:
https://my-engine/ovirt-engine/webadmin/?locale=en_US#disks-general;id=d9a29187-f492-4a0d-aea2-7d5216c957d7
Adding another disk in sorted in the middle (while the vm is running):
# virsh -r dumpxml disk-mapping
...
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='qcow2' cache='none'
error_policy='stop' io='native' discard='unmap'/>
<source
dev='/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5'
index='3'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore type='block' index='5'>
<format type='qcow2'/>
<source
dev='/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/ad891316-b80a-4ab5-895b-400108bd0ca1'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
</backingStore>
<target dev='sda' bus='scsi'/>
<serial>40018b33-2b11-4d10-82e4-604a5b135fb2</serial>
<boot order='1'/>
<alias name='ua-40018b33-2b11-4d10-82e4-604a5b135fb2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/3600140594af345ed76d42058f2b1a454' index='2'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>b97e68b2-87ea-45ca-94fb-277d5b30baa2</serial>
<alias name='ua-b97e68b2-87ea-45ca-94fb-277d5b30baa2'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5' index='1'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>d9a29187-f492-4a0d-aea2-7d5216c957d7</serial>
<alias name='ua-d9a29187-f492-4a0d-aea2-7d5216c957d7'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/36001405b4d0c0b7544d47438b21296ef' index='7'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vdc' bus='virtio'/>
<serial>e801c2e4-dc2e-4c53-b17b-bf6de99f16ed</serial>
<alias name='ua-e801c2e4-dc2e-4c53-b17b-bf6de99f16ed'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00'
function='0x0'/>
</disk>
...
# virsh -r domblklist disk-mapping
Target Source
---------------------------------------------------------------------------------------------------------------------------------------------------------------
sdc -
sda
/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5
vda /dev/mapper/3600140594af345ed76d42058f2b1a454
vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5
vdc /dev/mapper/36001405b4d0c0b7544d47438b21296ef
In the guest:
# ls -lh /dev/disk/by-id/virtio-*
lrwxrwxrwx. 1 root root 9 Jan 6 09:42
/dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda
lrwxrwxrwx. 1 root root 9 Jan 6 09:42
/dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb
lrwxrwxrwx. 1 root root 9 Jan 6 09:51
/dev/disk/by-id/virtio-e801c2e4-dc2e-4c53-b -> ../../vdc
Shutdown VM and start it again
# virsh -r dumpxml disk-mapping
...
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='qcow2' cache='none'
error_policy='stop' io='native' discard='unmap'/>
<source
dev='/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5'
index='4'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore type='block' index='6'>
<format type='qcow2'/>
<source
dev='/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/ad891316-b80a-4ab5-895b-400108bd0ca1'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
</backingStore>
<target dev='sda' bus='scsi'/>
<serial>40018b33-2b11-4d10-82e4-604a5b135fb2</serial>
<boot order='1'/>
<alias name='ua-40018b33-2b11-4d10-82e4-604a5b135fb2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/3600140594af345ed76d42058f2b1a454' index='3'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>b97e68b2-87ea-45ca-94fb-277d5b30baa2</serial>
<alias name='ua-b97e68b2-87ea-45ca-94fb-277d5b30baa2'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/36001405b4d0c0b7544d47438b21296ef' index='2'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>e801c2e4-dc2e-4c53-b17b-bf6de99f16ed</serial>
<alias name='ua-e801c2e4-dc2e-4c53-b17b-bf6de99f16ed'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00'
function='0x0'/>
</disk>
<disk type='block' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop'
io='native' iothread='1'/>
<source dev='/dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5' index='1'>
<seclabel model='dac' relabel='no'/>
</source>
<backingStore/>
<target dev='vdc' bus='virtio'/>
<serial>d9a29187-f492-4a0d-aea2-7d5216c957d7</serial>
<alias name='ua-d9a29187-f492-4a0d-aea2-7d5216c957d7'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
</disk>
...
# virsh -r domblklist disk-mapping
Target Source
---------------------------------------------------------------------------------------------------------------------------------------------------------------
sdc -
sda
/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5
vda /dev/mapper/3600140594af345ed76d42058f2b1a454
vdb /dev/mapper/36001405b4d0c0b7544d47438b21296ef
vdc /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5
In the guest:
# ls -lh /dev/disk/by-id/virtio-*
lrwxrwxrwx. 1 root root 9 Jan 6 09:55
/dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda
lrwxrwxrwx. 1 root root 9 Jan 6 09:55
/dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb
lrwxrwxrwx. 1 root root 9 Jan 6 09:55
/dev/disk/by-id/virtio-e801c2e4-dc2e-4c53-b -> ../../vdc
Comparing to state before reboot:
# virsh -r domblklist disk-mapping
Target Source
---------------------------------------------------------------------------------------------------------------------------------------------------------------
sdc -
sda
/rhev/data-center/mnt/blockSD/84dc4e3c-00fd-4263-84e8-fc246eeee6e9/images/40018b33-2b11-4d10-82e4-604a5b135fb2/40f455c4-8c92-4f8f-91c2-991b0ddfc2f5
vda /dev/mapper/3600140594af345ed76d42058f2b1a454
vdb /dev/mapper/360014050058f2f8a0474dc7a8a7cc6a5
vdc /dev/mapper/36001405b4d0c0b7544d47438b21296ef
# ls -lh /dev/disk/by-id/virtio-*
lrwxrwxrwx. 1 root root 9 Jan 6 09:42
/dev/disk/by-id/virtio-b97e68b2-87ea-45ca-9 -> ../../vda
lrwxrwxrwx. 1 root root 9 Jan 6 09:42
/dev/disk/by-id/virtio-d9a29187-f492-4a0d-a -> ../../vdb
lrwxrwxrwx. 1 root root 9 Jan 6 09:51
/dev/disk/by-id/virtio-e801c2e4-dc2e-4c53-b -> ../../vdc
In the guest disks are mapped to the same device name.
It looks like libivrt domblklist is not correct - vdb and vdc are switched.
Peter, this expected?
Nir
>
> Joy
>
> On Wed, Dec 2, 2020 at 1:28 PM Nir Soffer <nsof...@redhat.com> wrote:
>>
>> On Wed, Dec 2, 2020 at 10:27 AM Joy Li <joooy...@gmail.com> wrote:
>> >
>> > Hi All,
>> >
>> > I'm facing the problem that after adding disks to guest VM, the device
>> > target path changed (My ovirt version is 4.3). For example:
>> >
>> > Before adding a disk:
>> >
>> > virsh # domblklist <vmname>
>> > Target Source
>> > ---------------------------------------------------------
>> > hdc -
>> > vda /dev/mapper/3600a09803830386546244a546d494f53
>> > vdb /dev/mapper/3600a09803830386546244a546d494f54
>> > vdc /dev/mapper/3600a09803830386546244a546d494f55
>> > vdd /dev/mapper/3600a09803830386546244a546d494f56
>> > vde /dev/mapper/3600a09803830386546244a546d494f57
>> > vdf /dev/mapper/3600a09803830386546244a546d494f58
>> >
>> > After adding a disk, and then shutdown and start the VM:
>> >
>> > virsh # domblklist <vmname>
>> > Target Source
>> > ---------------------------------------------------------
>> > hdc -
>> > vda /dev/mapper/3600a09803830386546244a546d494f53
>> > vdb /dev/mapper/3600a09803830386546244a546d494f54
>> > vdc /dev/mapper/3600a09803830386546244a546d494f6c
>> > vdd /dev/mapper/3600a09803830386546244a546d494f55
>> > vde /dev/mapper/3600a09803830386546244a546d494f56
>> > vdf /dev/mapper/3600a09803830386546244a546d494f57
>> > vdg /dev/mapper/3600a09803830386546244a546d494f58
>> >
>> > The devices' multipath doesn't map to the same target path as before, so
>> > in my VM the /dev/vdc doesn't point to the old
>> > /dev/mapper/3600a09803830386546244a546d494f55 anymore.
>> >
>> > Anybody knows how can I make the device path mapping fixed without being
>> > changed after adding or removing disks.
>>
>> Device nodes are not stable, and oVirt cannot guarantee that you will
>> get the same
>> node in the guest in all runs.
>>
>> You should use /dev/disk/by-id/xxx links to located devices, and blkid to
>> create
>> fstab mounts that do not depend on node names.
>>
>> Regardless, oVirt try to keep devices stable as possible. Do you know
>> how to reproduce
>> this issue reliably?
>>
>> Nir
>>
