Improve and extend the qemu functions that save and restore VM state so a guest may be suspended and resumed with minimal pause time. qemu may be updated to a new version in between.
The first set of patches adds the cprsave and cprload commands to save and restore VM state, and allow the host kernel to be updated and rebooted in between. The VM must create guest RAM in a persistent shared memory file, such as /dev/dax0.0 or persistant /dev/shm PKRAM as proposed in https://lore.kernel.org/lkml/1588812129-8596-1-git-send-email-anthony.yzn...@oracle.com/ cprsave stops the VCPUs and saves VM device state in a simple file, and thus supports any type of guest image and block device. The caller must not modify the VM's block devices between cprsave and cprload. cprsave and cprload support guests with vfio devices if the caller first suspends the guest by issuing guest-suspend-ram to the qemu guest agent. The guest drivers suspend methods flush outstanding requests and re- initialize the devices, and thus there is no device state to save and restore. 1 savevm: add vmstate handler iterators 2 savevm: VM handlers mode mask 3 savevm: QMP command for cprsave 4 savevm: HMP Command for cprsave 5 savevm: QMP command for cprload 6 savevm: HMP Command for cprload 7 savevm: QMP command for cprinfo 8 savevm: HMP command for cprinfo 9 savevm: prevent cprsave if memory is volatile 10 kvmclock: restore paused KVM clock 11 cpu: disable ticks when suspended 12 vl: pause option 13 gdbstub: gdb support for suspended state The next patches add a restart method that eliminates the persistent memory constraint, and allows qemu to be updated across the restart, but does not allow host reboot. Anonymous memory segments used by the guest are preserved across a re-exec of qemu, mapped at the same VA, via a proposed madvise(MADV_DOEXEC) option in the Linux kernel. See https://lore.kernel.org/lkml/1595869887-23307-1-git-send-email-anthony.yzn...@oracle.com/ 14 savevm: VMS_RESTART and cprsave restart 15 vl: QEMU_START_FREEZE env var 16 oslib: add qemu_clr_cloexec 17 util: env var helpers 18 osdep: import MADV_DOEXEC 19 memory: ram_block_add cosmetic changes 20 vl: add helper to request re-exec 21 exec, memory: exec(3) to restart 22 char: qio_channel_socket_accept reuse fd 23 char: save/restore chardev socket fds 24 ui: save/restore vnc socket fds 25 char: save/restore chardev pty fds 26 monitor: save/restore QMP negotiation status 27 vhost: reset vhost devices upon cprsave 28 char: restore terminal on restart The next patches extend the restart method to save and restore vfio-pci state, eliminating the requirement for a guest agent. The vfio container, group, and device descriptors are preserved across the qemu re-exec. 29 pci: export pci_update_mappings 30 vfio-pci: save and restore 31 vfio-pci: trace pci config 32 vfio-pci: improved tracing Here is an example of updating qemu from v4.2.0 to v4.2.1 using "cprload restart". The software update is performed while the guest is running to minimize downtime. window 1 | window 2 | # qemu-system-x86_64 ... | QEMU 4.2.0 monitor - type 'help' ... | (qemu) info status | VM status: running | | # yum update qemu (qemu) cprsave /tmp/qemu.sav restart | QEMU 4.2.1 monitor - type 'help' ... | (qemu) info status | VM status: paused (prelaunch) | (qemu) cprload /tmp/qemu.sav | (qemu) info status | VM status: running | Here is an example of updating the host kernel using "cprload reboot" window 1 | window 2 | # qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...| QEMU 4.2.1 monitor - type 'help' ... | (qemu) info status | VM status: running | | # yum update kernel-uek (qemu) cprsave /tmp/qemu.sav restart | | # systemctl kexec | kexec_core: Starting new kernel | ... | | # qemu-system-x86_64 ...mem-path=/dev/dax0.0 ...| QEMU 4.2.1 monitor - type 'help' ... | (qemu) info status | VM status: paused (prelaunch) | (qemu) cprload /tmp/qemu.sav | (qemu) info status | VM status: running | Mark Kanda (5): char: qio_channel_socket_accept reuse fd char: save/restore chardev socket fds ui: save/restore vnc socket fds monitor: save/restore QMP negotiation status vhost: reset vhost devices upon cprsave Steve Sistare (27): savevm: add vmstate handler iterators savevm: VM handlers mode mask savevm: QMP command for cprsave savevm: HMP Command for cprsave savevm: QMP command for cprload savevm: HMP Command for cprload savevm: QMP command for cprinfo savevm: HMP command for cprinfo savevm: prevent cprsave if memory is volatile kvmclock: restore paused KVM clock cpu: disable ticks when suspended vl: pause option gdbstub: gdb support for suspended state savevm: VMS_RESTART and cprsave restart vl: QEMU_START_FREEZE env var oslib: add qemu_clr_cloexec util: env var helpers osdep: import MADV_DOEXEC memory: ram_block_add cosmetic changes vl: add helper to request re-exec exec, memory: exec(3) to restart char: save/restore chardev pty fds char: restore terminal on restart pci: export pci_update_mappings vfio-pci: save and restore vfio-pci: trace pci config vfio-pci: improved tracing MAINTAINERS | 7 ++ accel/kvm/kvm-all.c | 8 +- accel/kvm/trace-events | 3 +- chardev/char-pty.c | 38 +++++-- chardev/char-socket.c | 35 ++++++ chardev/char-stdio.c | 7 ++ chardev/char.c | 16 +++ exec.c | 88 +++++++++++++-- gdbstub.c | 11 +- hmp-commands.hx | 46 ++++++++ hw/i386/kvm/clock.c | 6 +- hw/pci/msix.c | 1 + hw/pci/pci.c | 17 +-- hw/pci/trace-events | 5 +- hw/vfio/common.c | 115 ++++++++++++++++---- hw/vfio/pci.c | 179 ++++++++++++++++++++++++++++++- hw/vfio/platform.c | 2 +- hw/vfio/trace-events | 11 +- hw/virtio/vhost.c | 12 +++ include/chardev/char.h | 8 ++ include/exec/memory.h | 4 + include/hw/pci/pci.h | 2 + include/hw/vfio/vfio-common.h | 4 +- include/io/channel-socket.h | 3 +- include/migration/register.h | 3 + include/migration/vmstate.h | 11 ++ include/monitor/hmp.h | 3 + include/qemu/cutils.h | 1 + include/qemu/env.h | 31 ++++++ include/qemu/osdep.h | 8 ++ include/sysemu/sysemu.h | 10 ++ io/channel-socket.c | 12 ++- io/net-listener.c | 4 +- migration/block.c | 1 + migration/migration.c | 4 +- migration/ram.c | 1 + migration/savevm.c | 237 ++++++++++++++++++++++++++++++++++++----- migration/savevm.h | 4 +- monitor/hmp-cmds.c | 28 +++++ monitor/qmp-cmds.c | 16 +++ monitor/qmp.c | 42 ++++++++ qapi/migration.json | 35 ++++++ qapi/pragma.json | 1 + qemu-options.hx | 9 ++ scsi/qemu-pr-helper.c | 2 +- softmmu/vl.c | 65 ++++++++++- tests/qtest/tpm-emu.c | 2 +- tests/test-char.c | 2 +- tests/test-io-channel-socket.c | 4 +- trace-events | 2 + ui/vnc.c | 153 +++++++++++++++++++++----- util/Makefile.objs | 2 +- util/env.c | 132 +++++++++++++++++++++++ util/oslib-posix.c | 9 ++ util/oslib-win32.c | 4 + 55 files changed, 1331 insertions(+), 135 deletions(-) create mode 100644 include/qemu/env.h create mode 100644 util/env.c -- 1.8.3.1
