Based-on: 74aa0fb297 (migration: options incompatible with cpr) # peterx/migration-next
Hi, In this v6: - Minor fixes to 17/23 and 19/23 CI run: https://gitlab.com/farosas/qemu/-/pipelines/1195796010 Series structure ================ This series enables mapped-ram in steps: 0) Cleanups [1] 1) QIOChannel interfaces [2-6] 2) Mapped-ram format for precopy [7-11] 3) Multifd adaptation without packets [12-15] 4) Mapped-ram format for multifd [16-23] * below will be sent separately * 5) Direct-io generic support [TODO] 6) Direct-io for mapped-ram multifd with file: URI [TODO] 7) Fdset interface for mapped-ram multifd [TODO] About mapped-ram ================ Mapped-ram is a new stream format for the RAM section designed to supplement the existing ``file:`` migration and make it compatible with ``multifd``. This enables parallel migration of a guest's RAM to a file. The core of the feature is to map RAM pages to migration file offsets. This enables the ``multifd`` threads to write exclusively to those offsets even if the guest is constantly dirtying pages (i.e. live migration). Another benefit is that the resulting file will have a bounded size, since pages which are dirtied multiple times will always go to a fixed location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment restrictions. Latest numbers (unchanged from v4) ============== => guest: 128 GB RAM - 120 GB dirty - 1 vcpu in tight loop dirtying memory => host: 128 CPU AMD EPYC 7543 - 2 NVMe disks in RAID0 (8586 MiB/s) - xfs => pinned vcpus w/ NUMA shortest distances - average of 3 runs - results from query-migrate non-live | time (ms) pages/s mb/s MB/s -------------------+----------------------------------- file | 110512 256258 9549 1193 + bg-snapshot | 245660 119581 4303 537 -------------------+----------------------------------- mapped-ram | 157975 216877 6672 834 + multifd 8 ch. | 95922 292178 10982 1372 + direct-io | 23268 1936897 45330 5666 ------------------------------------------------------- live | time (ms) pages/s mb/s MB/s -------------------+----------------------------------- file | - - - - (file grew 4x the VM size) + bg-snapshot | 357635 141747 2974 371 -------------------+----------------------------------- mapped-ram | - - - - (no convergence in 5 min) + multifd 8 ch. | 230812 497551 14900 1862 + direct-io | 27475 1788025 46736 5842 ------------------------------------------------------- v5: https://lore.kernel.org/r/20240228152127.18769-1-faro...@suse.de v4: https://lore.kernel.org/r/20240220224138.24759-1-faro...@suse.de v3: https://lore.kernel.org/r/20231127202612.23012-1-faro...@suse.de v2: https://lore.kernel.org/r/20231023203608.26370-1-faro...@suse.de v1: https://lore.kernel.org/r/20230330180336.2791-1-faro...@suse.de Fabiano Rosas (20): migration/multifd: Cleanup multifd_recv_sync_main io: fsync before closing a file channel migration/qemu-file: add utility methods for working with seekable channels migration/ram: Introduce 'mapped-ram' migration capability migration: Add mapped-ram URI compatibility check migration/ram: Add outgoing 'mapped-ram' migration migration/ram: Add incoming 'mapped-ram' migration tests/qtest/migration: Add tests for mapped-ram file-based migration migration/multifd: Rename MultiFDSend|RecvParams::data to compress_data migration/multifd: Decouple recv method from pages migration/multifd: Allow multifd without packets migration/multifd: Allow receiving pages without packets migration/multifd: Add a wrapper for channels_created migration/multifd: Add outgoing QIOChannelFile support migration/multifd: Add incoming QIOChannelFile support migration/multifd: Prepare multifd sync for mapped-ram migration migration/multifd: Support outgoing mapped-ram stream format migration/multifd: Support incoming mapped-ram stream format migration/multifd: Add mapped-ram support to fd: URI tests/qtest/migration: Add a multifd + mapped-ram migration test Nikolay Borisov (3): io: add and implement QIO_CHANNEL_FEATURE_SEEKABLE for channel file io: Add generic pwritev/preadv interface io: implement io_pwritev/preadv for QIOChannelFile docs/devel/migration/features.rst | 1 + docs/devel/migration/mapped-ram.rst | 138 ++++++++++ include/exec/ramblock.h | 13 + include/io/channel.h | 83 ++++++ include/migration/qemu-file-types.h | 2 + include/qemu/bitops.h | 13 + io/channel-file.c | 69 +++++ io/channel.c | 58 ++++ migration/fd.c | 44 +++ migration/fd.h | 2 + migration/file.c | 149 +++++++++- migration/file.h | 8 + migration/migration.c | 56 +++- migration/multifd-zlib.c | 26 +- migration/multifd-zstd.c | 26 +- migration/multifd.c | 405 ++++++++++++++++++++++------ migration/multifd.h | 27 +- migration/options.c | 35 +++ migration/options.h | 1 + migration/qemu-file.c | 106 ++++++++ migration/qemu-file.h | 6 + migration/ram.c | 345 ++++++++++++++++++++++-- migration/ram.h | 1 + migration/savevm.c | 1 + migration/trace-events | 2 +- qapi/migration.json | 6 +- tests/qtest/migration-test.c | 127 +++++++++ 27 files changed, 1607 insertions(+), 143 deletions(-) create mode 100644 docs/devel/migration/mapped-ram.rst -- 2.35.3
