KreVet-Ka commented on issue #14508:
URL: https://github.com/apache/cloudstack/issues/14508#issuecomment-6473094296
### Additional findings from a production cluster: silent no-op detach when
the live domain XML has a legacy `/dev/drbdN` path
We operate the cluster where the original report was made, and we were able
to confirm the mechanism from our logs and hypervisors and assess how
widespread it is.
### **Environment:**
CloudStack 4.20.3.1 (upgraded from 4.20.3.0), KVM, libvirt 10.0.0, QEMU
8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18),
Linstor primary storage (DRBD 9, hypervisors are diskless clients).
### Root cause (as far as we can tell from code and logs)
- Before the Linstor plugin change (#12300) the plugin returned
`/dev/drbd{minor}` as the disk path. After it, the plugin returns
`/dev/drbd/by-res/cs-<volume path>/0`.
- A VM started **before** the upgrade keeps the old path in its **live**
libvirt XML (`<source dev='/dev/drbd1234'/>`). Live migration preserves it too
(we verified this across migrations); only a stop/start regenerates the XML
with the new path.
- On `detachVolume`, `KVMStorageProcessor.attachOrDetachDisk` looks for the
disk in the live XML by the exact path string. The strings differ even though
they point to the same block device (`/dev/drbd/by-res/cs-X/0` is a symlink to
`/dev/drbdN`), so the disk is not found. The agent logs the following and
**returns success**:
```
WARN [kvm.storage.KVMStorageProcessor] Could not find disk
[/dev/drbd/by-res/cs-<volume-path>/0] attached to VM instance with UUID
[<vm-uuid>]. We will set it as detached in the database to ensure consistency.
```
- No `detach-device` is issued, so QEMU never emits `DEVICE_DELETED`. The
API job succeeds, the DB sets `instance_id = NULL`, but the disk stays attached
in the guest and the DRBD device stays open in the QEMU process.
### How it manifests
1. The API reports success, but the disk is still attached inside the guest
and in `virsh domblklist`.
2. Attaching the same volume to a VM on a **different host** fails (`Wrong
medium type`): DRBD cannot promote a second Primary when `allow-two-primaries`
is off for that resource.
3. Re-attaching to the **same VM** fails with `target 'vdb' duplicated for
disk sources '/dev/drbdN' and '/dev/drbd/by-res/…'`.
4. Attaching it to **another VM on the same host succeeds**, so the same
DRBD device is open in two QEMU processes at once. In the reported case this
window lasted roughly an hour (the stale device was later removed out-of-band
from inside the guest). There is no protection against concurrent writers, so
this is a potential data corruption risk. We cannot prove or rule out actual
writes from the host side.
We reproduced steps 1 and 3 on a lab VM started before the upgrade (detach
returned success, WARN logged, disk still present in `virsh domblklist`).
### How widespread it is on our cluster
- More than half of the running domains still carry a legacy `/dev/drbdN`
path on at least one disk. All of them have a legacy root disk, i.e. they were
started before the upgrade.
- A smaller share of those also have legacy **data** disks; these are the
ones exposed to this bug. Root-only VMs are not exposed, since a root volume
cannot be detached while the VM is running.
- Kubernetes workers with CSI-managed PVCs are the worst case: the CSI
driver detaches and attaches volumes automatically whenever pods move, so users
cannot avoid the trigger.
- Our agent logs contain several past occurrences of the WARN above (the
logs we still have go back to early June), so this has been happening silently
since the upgrade.
Check on a host: `virsh domblklist <domain> --details | awk '$1=="block" &&
$2=="disk" {print $3, $4}' | grep -E ' /dev/drbd[0-9]+$'`
Happy to provide any logs you might need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
