As reported in the community forum [0] and then later by Thomas, who provided the relevant system logs, parallel migration with '--with-conntrack-state' of multiple VMs may currently lead to a crash upon handover:
> kvm: Unknown savevm section or instance 'dbus-vmstate/dbus-vmstate' 0. > Make sure that your current VM setup matches your saved VM setup, > including any hotplugged devices > kvm: load of migration failed: Invalid argument In particular, the following sequence (on my test node) pvesh create /nodes/pve9a1/qemu/104/dbus-vmstate --action start pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action start pvesh create /nodes/pve9a1/qemu/105/dbus-vmstate --action stop results in the wrong service being shut down (note the unexpected ID in the last line!): Dec 10 10:07:40 pve9a1 pvesh[30453]: starting dbus-vmstate helper for VM 104 Dec 10 10:07:40 pve9a1 systemd[1]: Starting [email protected] - PVE DBus VMState Helper (VM 104)... Dec 10 10:07:41 pve9a1 dbus-vmstate[30456]: pve-vmstate-104 listening on :1.55 Dec 10 10:07:41 pve9a1 systemd[1]: Started [email protected] - PVE DBus VMState Helper (VM 104). Dec 10 10:07:44 pve9a1 pvesh[30511]: starting dbus-vmstate helper for VM 105 Dec 10 10:07:44 pve9a1 systemd[1]: Starting [email protected] - PVE DBus VMState Helper (VM 105)... Dec 10 10:07:45 pve9a1 dbus-vmstate[30573]: pve-vmstate-105 listening on :1.58 Dec 10 10:07:45 pve9a1 systemd[1]: Started [email protected] - PVE DBus VMState Helper (VM 105). Dec 10 10:07:48 pve9a1 pvesh[30595]: stopping dbus-vmstate helper for VM 105 Dec 10 10:07:48 pve9a1 dbus-vmstate[30456]: shutting down gracefully .. Dec 10 10:07:48 pve9a1 systemd[1]: [email protected]: Deactivated successfully. So the dbus-vmstate object is removed from the wrong VM before loading the migration state. Note that the crash is still racy, because if the dbus-vmstate is removed on the source side for the same wrong VM before the migration handover, the QEMU objects for both instances will still match. To fix the issue, use the dbus_call_method() helper. Like, this the owner is respected even if there are multiple (queued) owners on the DBus. [0]: https://forum.proxmox.com/threads/176821/post-820775 Fixes: dc76a590 ("fix #5180: migrate: integrate helper for live-migrating conntrack info") Reported-by: Thomas Lamprecht <[email protected]> Signed-off-by: Fiona Ebner <[email protected]> --- Changes in v3: * split patch. src/PVE/QemuServer/DBusVMState.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/PVE/QemuServer/DBusVMState.pm b/src/PVE/QemuServer/DBusVMState.pm index 05dd1bcd..480d9f70 100644 --- a/src/PVE/QemuServer/DBusVMState.pm +++ b/src/PVE/QemuServer/DBusVMState.pm @@ -127,7 +127,8 @@ sub qemu_del_dbus_vmstate { $num_entries = eval { dbus_get_property($object, 'com.proxmox.VMStateHelper', 'NumMigratedEntries'); }; - eval { $object->Quit() }; + # Quit() does QMP object-del which has a timeout of 60 seconds + eval { dbus_call_method($object, 'com.proxmox.VMStateHelper', 'Quit', [], 70); }; if (my $err = $@) { syslog('warn', "failed to call quit on dbus-vmstate for VM $vmid: $err\n") if !$params{quiet}; -- 2.47.3 _______________________________________________ pve-devel mailing list [email protected] https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
