This even happens with the most common image-file backed disks which further simplifies the repro:
$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc $ virsh detach-disk h vdc ** Description changed: Repro: - # Prep a backing disk - $ sudo fallocate -l 100M /tmp/zfsbase1 - $ sudo fallocate -l 100M /tmp/zfsbase2 - $ sudo zpool create zfsmirrortest mirror /tmp/zfsbase1 /tmp/zfsbase2 - $ sudo zfs create -V 20M zfsmirrortest/vol1 - $ cat > disk-zfs.xml << EOF - <disk type='block' device='disk'> - <driver name='qemu' type='raw'/> - <source dev='/dev/zvol/zfsmirrortest/vol1'/> - <target dev='vdc' bus='virtio'/> - </disk> - EOF - # Get a guest + #1 Get a guest $ uvt-kvm create --disk 5 --password=ubuntu h release=hirsute arch=s390x label=daily $ uvt-kvm wait h release=hirsute arch=s390x label=daily - # Attach and Detach disk - $ virsh attach-device h disk-zfs.xml - Device attached successfully - $ virsh detach-device h disk-zfs.xml - Device detached successfully + + #2 Attach and Detach disk + $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M + $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc + $ virsh detach-disk h vdc From libvirts POV it is gone at this point $ virsh domblklist h - Target Source + Target Source ------------------------------------------------------------------ - vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow - vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow - + vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow + vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow But the guest thinks still it is present $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk - ... - vdc 252:32 0 20M 0 disk + ... + vdc 252:32 0 20M 0 disk This even remains a while after (not a race). Any access to it in the guest will hang (as you'd expect of a non-existing blockdev) 4 0 1758 1739 20 0 12140 4800 - S+ pts/0 0:00 | \_ sudo mkfs.ext4 /dev/vdc 4 0 1759 1758 20 0 6924 1044 - D+ pts/0 0:00 | \_ mkfs.ext4 /dev/vdc - - The result above was originally found with hirsute-guest@hirsute-host on s390x + The result above was originally found with hirsute-guest@hirsute-host on + s390x I do NOT see the same with groovy-guest@hirsute-host on s390x I DO see the same with hirsute-guest@groovy-host on s390x - => Guest version dependent not Host/Hipervisor dependent + => Guest version dependent not Host/Hipervisor dependent I DO see the same with ZFS disks AND LVM disks being added&removed - => not type dependent + => not type dependent I do NOT see the same on x86. - => Arch dependent ?? + => Arch dependent ?? ... the evidence slowly points towards an issue in the guest, damn we are so close to release - but non-fully detaching disks are critical in my POV :-/ Filing this as-is for awareness, but certainly this will need more debugging. Unsure where this is going to eventually I'll now file it for kernel/udev/systemd. If there are any known issues/components that are related let me know please! ** Tags added: apport-collected uec-images ** Description changed: Repro: #1 Get a guest $ uvt-kvm create --disk 5 --password=ubuntu h release=hirsute arch=s390x label=daily $ uvt-kvm wait h release=hirsute arch=s390x label=daily #2 Attach and Detach disk $ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test.qcow2 10M $ virsh attach-disk h /var/lib/libvirt/images/test.qcow2 vdc $ virsh detach-disk h vdc From libvirts POV it is gone at this point $ virsh domblklist h Target Source ------------------------------------------------------------------ vda /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs.qcow vdb /var/lib/uvtool/libvirt/images/hirsute-2nd-zfs-ds.qcow But the guest thinks still it is present $ uvt-kvm ssh --insecure hirsute-2nd-zfs lsblk ... vdc 252:32 0 20M 0 disk This even remains a while after (not a race). Any access to it in the guest will hang (as you'd expect of a non-existing blockdev) 4 0 1758 1739 20 0 12140 4800 - S+ pts/0 0:00 | \_ sudo mkfs.ext4 /dev/vdc 4 0 1759 1758 20 0 6924 1044 - D+ pts/0 0:00 | \_ mkfs.ext4 /dev/vdc The result above was originally found with hirsute-guest@hirsute-host on s390x I do NOT see the same with groovy-guest@hirsute-host on s390x I DO see the same with hirsute-guest@groovy-host on s390x => Guest version dependent not Host/Hipervisor dependent I DO see the same with ZFS disks AND LVM disks being added&removed => not type dependent I do NOT see the same on x86. => Arch dependent ?? ... the evidence slowly points towards an issue in the guest, damn we are so close to release - but non-fully detaching disks are critical in my POV :-/ Filing this as-is for awareness, but certainly this will need more debugging. Unsure where this is going to eventually I'll now file it for kernel/udev/systemd. If there are any known issues/components that are related let me know please! + --- + ProblemType: Bug + AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory + AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' + ApportVersion: 2.20.11-0ubuntu65 + Architecture: s390x + ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' + CRDA: N/A + CasperMD5CheckResult: unknown + DistroRelease: Ubuntu 21.04 + IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' + Lspci: + + Lspci-vt: -[0000:00]- + Lsusb: Error: command ['lsusb'] failed with exit code 1: + Lsusb-t: Error: command ['lsusb', '-t'] failed with exit code 1: /sys/bus/usb/devices: No such file or directory + Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1: + Package: udev + PackageArchitecture: s390x + PciMultimedia: + + ProcFB: + + ProcKernelCmdLine: root=LABEL=cloudimg-rootfs + ProcVersionSignature: User Name 5.11.0-14.15-generic 5.11.12 + RelatedPackageVersions: + linux-restricted-modules-5.11.0-14-generic N/A + linux-backports-modules-5.11.0-14-generic N/A + linux-firmware N/A + RfKill: Error: [Errno 2] No such file or directory: 'rfkill' + Tags: hirsute uec-images + Uname: Linux 5.11.0-14-generic s390x + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: adm audio cdrom dialout dip floppy lxd netdev plugdev sudo video + _MarkForUpload: True + acpidump: -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1925211 Title: Hot-unplug of disks leaves broken block devices around in Hirsute To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1925211/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs