wzzirro opened a new issue, #14508: URL: https://github.com/apache/cloudstack/issues/14508
### problem Since #12300 (4.20.3.0), the Linstor adaptor returns `/dev/drbd/by-res/<rsc>/0` as the device path of a volume. If a data disk was attached before the upgrade, and its VM has not been stopped/started via CloudStack since, the live domain XML still has the raw DRBD minor path, e.g. `/dev/drbd1069`. This is the same legacy-path situation that #13538 fixed for NAS backups. On detach, `attachOrDetachDisk(attach=false)` looks the disk up in the domain XML by an exact path comparison: https://github.com/apache/cloudstack/blob/e742aadfc97e84b65f4b24052f57f5ee0bd2c849/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java#L1414-L1425 `/dev/drbd/by-res/cs-<uuid>/0` does not equal `/dev/drbd1069`, so the lookup fails and the agent logs ``` Could not find disk [/dev/drbd/by-res/cs-<uuid>/0] attached to VM instance with UUID [...]. We will set it as detached in the database to ensure consistency. ``` The agent then returns without sending `detach-device`. `dettachVolume` calls `disconnectPhysicalDisk` and answers success. The volume becomes `Ready` with no VM in the DB. However, the disk is still in the domain, QEMU keeps the DRBD device open, and the guest never gets an unplug request. What we saw after such a "successful" detach (VM-A holds the stale disk): 1. Attach to a VM on another host fails: ``` org.libvirt.LibvirtException: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/dev/drbd/by-res/cs-<uuid>/0': Wrong medium type ``` 2. Attach back to VM-A fails: ``` org.libvirt.LibvirtException: XML error: target 'vdb' duplicated for disk sources '/dev/drbd1069' and '/dev/drbd/by-res/cs-<uuid>/0' ``` 3. Attach to another VM, apparently on the same host as VM-A, **succeeds**. The same DRBD device is then attached to two running VMs at once. We had two volumes in this state. The stale copies were not mounted, but nothing prevents both guests from writing to the device. Guest-side workaround: in the VM that still has the disk, run `echo 0 > /sys/bus/pci/slots/<N>/power`. The guest ejects the device within a second, libvirt drops it from the domain, and the volume can be attached elsewhere. This also shows that ACPI hotplug works in the guest: it just never received a request. Note: we are a tenant and have no access to the agent logs. The log line above is from reading the code. The rest (API events and results, libvirt errors returned by attachVolume, guest state) we observed directly. ### versions - CloudStack 4.20.3.x, KVM, Linstor primary storage. The volume was attached before the cloud was upgraded from a release without #12300. - QEMU / libvirt / DRBD / LINSTOR versions: unknown to us. - Guests: Ubuntu 24.04, kernel 6.8, PCI (ACPI) hotplug. Volumes are attached/detached by the leaseweb cloudstack-csi-driver 0.10.0, which only calls `attachVolume` / `detachVolume`. ### The steps to reproduce the bug 1. On a release without #12300 (e.g. 4.20.2.0) with Linstor primary storage, attach a data volume to a running VM. 2. Upgrade management and KVM agents to 4.20.3.x. Do not stop/start the VM. 3. `detachVolume` the volume: the job succeeds and the volume has no VM. 4. On the host, `virsh domblklist <vm>` still shows the disk as `/dev/drbdNNNN`, and the guest still sees it. 5. `attachVolume` to a VM on another host fails with `Wrong medium type`. To the same VM it fails with `target ... duplicated`. To a VM on the same host it succeeds. ### What to do about it? - When looking up the disk to detach, compare resolved paths (`/dev/drbd/by-res/<rsc>/0` is a udev symlink to `/dev/drbdNNNN`) or match by disk serial (`diskUuidToSerial(vol.getUuid())`), not by the raw source string. - Do not report a successful detach when no disk matched the path but a disk with the volume's serial is still in the domain. That is the case where "set it as detached in the database to ensure consistency" is unsafe. - Optionally, refuse an attach while the Linstor resource is still in use by another domain, outside of live migration. Related: #12300 (path change), #13538 (same legacy path, fixed for NAS backup), #14214 (orphaned disk in libvirt after detach, different cause). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
