On Tue Jun 3, 2025 at 11:35 AM CEST, Stefan Hanreich wrote: > Gave this a spin on my test cluster, and it worked as advertised. As > already mentioned and talked off-list we should imo add documentation > for this setting and mention the current restrictions (no migration of > NAT entries). We might also want to mention that only CT entries created > *after* the firewall is enabled for a VM get migrated, although that is > probably rather uncommon.
Thanks for the (pretty extensive, as also talked off-list) testing! I'll definitely add documentation in the next revision, mentioning especially all the limitations. > > Other than that consider this: > > Tested-by: Stefan Hanreich <s.hanre...@proxmox.com> > > I've also mainly looked at the firewall + pve-manager code, maybe > someone with more knowledge about QEMU can chime in on the DBus server > parts. Consider the firewall and ui patches: > > Reviewed-by: Stefan Hanreich <s.hanre...@proxmox.com> > > > On 4/24/25 13:19, Christoph Heiss wrote: >> Fixes #5180 [0]. >> >> This implements migration of per-VM conntrack state on live-migration. >> >> The core of the implementation are in patch #7 & #8. See there for more >> details. >> >> Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique >> VMID. This is needed later on to filter conntrack entries for the >> migration. These three patches can be applied independently, >> CONNMARK'ing traffic does not have any visible impact. >> >> Currently, remote/inter-cluster migration is not supported and indicated >> to the user with a warning. See also patch #8 for a bit more in-depth >> explanation. >> >> Needed dependency bumps between packages are indicated in the notes >> appropriately. >> >> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180 >> >> Testing >> ======= >> >> I've primarily tested intra-cluster live-migrations, with both the >> iptables-based and nftables-based firewall), using the reproducer as >> described in #5180. I further verified that the D-Bus servers get >> started as expected and are _always_ stopped, even in the case of some >> migration error. >> >> Finally, I also checked using `conntrack -L -m <vmid>` tool that the >> conntrack entries are >> a) added/updated on the target node and >> b) removed from the source node afterwards >> >> Also tested was the migration from/to an "old" (unpatched) node, which >> results in the issue as per #5180 & appropriate warnings in the UI. >> >> For remote migrations, only tested that the warning is logged as >> expected. >> >> History >> ======= >> >> v1: >> https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.he...@proxmox.com/ >> >> Changes v1 -> v2: >> * rebased as necessary >> * "un-rfc'd" firewall conntrack flushing patches >> * use an instanced systemd service instead of fork+exec for the >> pve-dbus-vmstate helper >> >> Diffstat >> ======== >> >> pve-firewall: >> >> Christoph Heiss (2): >> firewall: add connmark rule with VMID to all guest chains >> firewall: helpers: add sub for flushing conntrack entries by mark >> >> debian/control | 3 ++- >> src/PVE/Firewall.pm | 7 +++++-- >> src/PVE/Firewall/Helpers.pm | 11 +++++++++++ >> 3 files changed, 18 insertions(+), 3 deletions(-) >> >> proxmox-firewall: >> >> Christoph Heiss (1): >> firewall: add connmark rule with VMID to all guest chains >> >> proxmox-firewall/src/firewall.rs | 14 +++- >> .../integration_tests__firewall.snap | 84 +++++++++++++++++++ >> proxmox-nftables/src/expression.rs | 9 ++ >> proxmox-nftables/src/statement.rs | 10 ++- >> 4 files changed, 114 insertions(+), 3 deletions(-) >> >> proxmox-ve-rs: >> >> Christoph Heiss (1): >> config: guest: allow access to raw Vmid value >> >> proxmox-ve-config/src/guest/types.rs | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> qemu-server: >> >> Christoph Heiss (5): >> qmp helpers: allow passing structured args via qemu_objectadd() >> api2: qemu: add module exposing node migration capabilities >> fix #5180: libexec: add QEMU dbus-vmstate daemon for migrating >> conntrack >> fix #5180: migrate: integrate helper for live-migrating conntrack info >> migrate: flush old VM conntrack entries after successful migration >> >> Makefile | 7 +- >> PVE/API2/Qemu.pm | 72 +++++++++++ >> PVE/API2/Qemu/Makefile | 2 +- >> PVE/API2/Qemu/Migration.pm | 46 +++++++ >> PVE/CLI/qm.pm | 5 + >> PVE/QemuMigrate.pm | 69 ++++++++++ >> PVE/QemuServer.pm | 6 + >> PVE/QemuServer/DBusVMState.pm | 120 ++++++++++++++++++ >> PVE/QemuServer/Makefile | 1 + >> PVE/QemuServer/QMPHelpers.pm | 4 +- >> dbus-vmstate/Makefile | 7 ++ >> dbus-vmstate/dbus-vmstate | 168 +++++++++++++++++++++++++ >> dbus-vmstate/org.qemu.VMState1.conf | 11 ++ >> dbus-vmstate/pve-dbus-vmstate@.service | 10 ++ >> debian/control | 7 +- >> 15 files changed, 530 insertions(+), 5 deletions(-) >> create mode 100644 PVE/API2/Qemu/Migration.pm >> create mode 100644 PVE/QemuServer/DBusVMState.pm >> create mode 100644 dbus-vmstate/Makefile >> create mode 100755 dbus-vmstate/dbus-vmstate >> create mode 100644 dbus-vmstate/org.qemu.VMState1.conf >> create mode 100644 dbus-vmstate/pve-dbus-vmstate@.service >> >> pve-manager: >> >> Christoph Heiss (4): >> api2: capabilities: explicitly import CPU capabilities module >> api2: capabilities: proxy index endpoints to respective nodes >> api2: capabilities: expose new qemu/migration endpoint >> ui: window: Migrate: add checkbox for migrating VM conntrack state >> >> PVE/API2/Capabilities.pm | 9 +++++ >> www/manager6/window/Migrate.js | 73 ++++++++++++++++++++++++++++++++-- >> 2 files changed, 78 insertions(+), 4 deletions(-) >> _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
