This is an RFC series for fast migration of confidential guests using an
in-guest migration helper that lives in OVMF. QEMU VM live migration
needs to read source VM's RAM and write it in the target VM; this
mechanism doesn't work when the guest memory is encrypted or QEMU is
prevented from reading it in another way. In order to support live
migration in such scenarios, we introduce an in-guest migration helper
which can securely extract RAM content from the guest in order to send
it to the target. The migration helper is implemented as part of the
VM's firmware in OVMF.
We've implemented and tested this on AMD SEV, but expect most of the
processes can be used with other technologies that prevent direct access
of hypervisor to the guest's memory. Specifically, we don't use SEV's
PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all;
but note that the mirror VM relies on KVM_CAP_VM_COPY_ENC_CONTEXT_FROM
to shared the SEV ASID with the main VM.
Corresponding RFC patches for OVMF have been posted by Tobin
Feldman-Fitzthum on edk2-devel [1]. Those include the crux of the
migration helper: a mailbox protocol over a shared memory page which
allows communication between QEMU and the migration helper. In the
source VM this is used to read a page and encrypt it for transport; in
the target it is used to decrypt the incoming page and storing the
content in the correct address in the guest memory. All encryption and
decryption operations occur inside the trusted context in the VM, and
therefore the VM's memory plaintext content is never accessible to the
hosts participating in the migration.
In order to allow OVMF to run the migration helper in parallel to the
guest OS, we use a mirror VM [3], which shares the same memory mapping
and SEV ASID as the main VM but has its own run loop. To start the
mirror vcpu and the migration handler, we added a temporary
start-migration-handler QMP command; this will be removed in a future
version to run as part of the migrate QMP command.
In the target VM we need the migration handler running to receive
incoming RAM pages; to achieve that, we boot the VM into OVMF with a
special fw_cfg value that causes OVMF to not boot the guest OS; we then
allow QEMU to receive an incoming migration by issuing a new
start-migrate-incoming QMP command.
The confidential RAM migration requires checking whether a given guest
RAM page is encrypted or not. This is achieved using SEV shared regions
list tracking, which is implemented as part the SEV live migration patch
series [2]. This feature tracks hypercalls from OVMF and guest Linux to
report changes of page encryption status so that QEMU has an up-to-date
view of which memory regions are shared and which are encrypted.
We left a few unfinished edges in this RFC but decided to publish it to
start the commmunity discussion. TODOs:
1. QMP commands start-migration-handler and start-migrate-incoming are
developer tools and should be performed automatically.
2. The entry point address of the in-guest migration handler and its GDT
are currently hard-coded in QEMU (patch 8); instead they should be
discovered using pc_system_ovmf_table_find. Same applies for the
mailbox address (patch 1).
3. For simplicity, this patch series forces the use of the
guest-assisted migration instead of the SEV PSP-based migration.
Ideally we might want the user to choose the desired mode using
migrate-set-parameters or a similar mechanism.
4. There is currently no discovery protocol between QEMU and OVMF to
verify that OVMF indeed supports in-guest migration handler.
List of patches in this series:
1-3: introduce new confidtial RAM migration functions which communicate
with the migration helper.
4-6: use the new MH communication functions when migrating encrypted RAM
pages
7-9: allow starting migration handler on mirror vcpu with QMP command
start-migration-handler
10: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
11: fix devices issues when loading state into a live VM
12: add documentation
This patch series is based on top of:
1. Add SEV guest live migration support, from Ashish Kalra [2]
2. Support for mirror VM, from Ashish Kalra [3]
[1] https://edk2.groups.io/g/devel/message/79517
[2]
https://lore.kernel.org/qemu-devel/cover.1628076205.git.ashish.ka...@amd.com/
[3]
https://lore.kernel.org/qemu-devel/cover.1629118207.git.ashish.ka...@amd.com/
Changes from RFC v1:
- Use the an SEV mirror VM for the migation handler (instead of
auxilliary vcpus)
RFC v1:
https://lore.kernel.org/qemu-devel/20210302204822.81901-1-dovmu...@linux.vnet.ibm.com/
Dov Murik (12):
migration: Add helpers to save confidential RAM
migration: Add helpers to load confidential RAM
migration: Introduce gpa_inside_migration_helper_shared_area
migration: Save confidential guest RAM using migration helper
migration: Load confidential guest RAM using migration helper
migration: Skip ROM, non-RAM, and vga.vram memory region during RAM
migration
i386/kvm: Exclude mirror vcpu in kvm_synchronize_all_tsc
migration: Allow resetting the mirror vcpu to the MH entry point
migration: Add QMP command start-migration-handler
migration: Add start-migrate-incoming QMP command
hw/isa/lpc_ich9: Allow updating an already-running VM
docs: Add confidential guest live migration documentation
docs/confidential-guest-live-migration.rst | 145 +++++++++
docs/confidential-guest-support.txt | 5 +
docs/index.rst | 1 +
qapi/migration.json | 38 +++
include/sysemu/sev.h | 1 +
migration/confidential-ram.h | 23 ++
hw/isa/lpc_ich9.c | 3 +-
migration/confidential-ram.c | 339 +++++++++++++++++++++
migration/migration.c | 29 ++
migration/ram.c | 133 +++++++-
target/i386/kvm/kvm.c | 4 +-
migration/meson.build | 2 +-
migration/trace-events | 4 +
13 files changed, 714 insertions(+), 13 deletions(-)
create mode 100644 docs/confidential-guest-live-migration.rst
create mode 100644 migration/confidential-ram.h
create mode 100644 migration/confidential-ram.c
--
2.20.1
