This patch series adds the live migration cpr-exec mode. The new user-visible interfaces are: * cpr-exec (MigMode migration parameter) * cpr-exec-command (migration parameter)
cpr-exec mode is similar in most respects to cpr-transfer mode, with the primary difference being that old QEMU directly exec's new QEMU. The user specifies the command to exec new QEMU in the migration parameter cpr-exec-command. Why? In a containerized QEMU environment, cpr-exec reuses an existing QEMU container and its assigned resources. By contrast, cpr-transfer mode requires a new container to be created on the same host as the target of the CPR operation. Resources must be reserved for the new container, while the old container still reserves resources until the operation completes. Avoiding over commitment requires extra work in the management layer. This is one reason why a cloud provider may prefer cpr-exec. A second reason is that the container may include agents with their own connections to the outside world, and such connections remain intact if the container is reused. How? cpr-exec preserves descriptors across exec by clearing the CLOEXEC flag, and by sending the unique name and value of each descriptor to new QEMU via CPR state. CPR state cannot be sent over the normal migration channel, because devices and backends are created prior to reading the channel, so this mode sends CPR state over a second migration channel that is not visible to the user. New QEMU reads the second channel prior to creating devices or backends. The exec itself is trivial. After writing to the migration channels, the migration code calls a new main-loop hook to perform the exec. Example: In this example, we simply restart the same version of QEMU, but in a real scenario one would use a new QEMU binary path in cpr-exec-command. # qemu-kvm -monitor stdio -object memory-backend-memfd,id=ram0,size=1G -machine memory-backend=ram0 -machine aux-ram-share=on ... QEMU 10.1.50 monitor - type 'help' for more information (qemu) info status VM status: running (qemu) migrate_set_parameter mode cpr-exec (qemu) migrate_set_parameter cpr-exec-command qemu-kvm ... -incoming file:vm.state (qemu) migrate -d file:vm.state (qemu) QEMU 10.1.50 monitor - type 'help' for more information (qemu) info status VM status: running Steve Sistare (9): migration: multi-mode notifier migration: add cpr_walk_fd oslib: qemu_clear_cloexec vl: helper to request exec migration: cpr-exec-command parameter migration: cpr-exec save and load migration: cpr-exec mode migration: cpr-exec docs vfio: cpr-exec mode docs/devel/migration/CPR.rst | 103 ++++++++++++++++++++++++- qapi/migration.json | 46 ++++++++++- include/migration/cpr.h | 9 +++ include/migration/misc.h | 12 +++ include/qemu/osdep.h | 9 +++ include/system/runstate.h | 3 + hw/vfio/container.c | 3 +- hw/vfio/cpr-iommufd.c | 3 +- hw/vfio/cpr-legacy.c | 9 ++- hw/vfio/cpr.c | 13 ++-- migration/cpr-exec.c | 168 +++++++++++++++++++++++++++++++++++++++++ migration/cpr.c | 39 +++++++++- migration/migration-hmp-cmds.c | 25 ++++++ migration/migration.c | 70 +++++++++++++---- migration/options.c | 14 ++++ migration/ram.c | 1 + migration/vmstate-types.c | 8 ++ system/runstate.c | 29 +++++++ util/oslib-posix.c | 9 +++ util/oslib-win32.c | 4 + hmp-commands.hx | 2 +- migration/meson.build | 1 + migration/trace-events | 1 + 23 files changed, 548 insertions(+), 33 deletions(-) create mode 100644 migration/cpr-exec.c -- 1.8.3.1
