applied series On Mon, Jun 07, 2021 at 05:35:23PM +0200, Stefan Reiter wrote: > This series is the third/fourth attempt[0] at substantially improving > live-restore performance. This time, a fully async- and concurrency safe LRU > cache is implemented, and a new CachedChunkReader is used to provide lock-free > reading from a remote chunk source. The big performance improvements come from > removing the global Mutex synchronising all read_image_at requests in the QEMU > library and the shared LRU cache. > > The idea of using 'mmap' to alleviate memory pressure was dropped in favor of > using a shared cache between drives. This provides about the same performance > improvement in typical live-restore scenarios, but with a fixed memory > requirement, independant of drives. Adding 'mmap' caching is doable, but the > Rust code really doesn't look very nice with it, and I think it's not > necessary > in this version... > > Anyway, I figured out how to better benchmark the changes too, and can now > reproduce the performance gains very reliably. Below is a (gnu)plot of running > 40 live-restores of a Windows 10 VM with a single 32GB disk, first with the > currently shipping libproxmox-backup-qemu0 (A) and then with the patched one > from this series (B). Testing was done with QEMU 6.0. > > > v2: > * address Wolfgang's review comments, mostly style stuff > > > seconds > 240 +-----------------------------------------------------------------------+ > | + + + + + + A + | > 230 |-+ A A A +-| > | A A A AA A A A A | > 220 |-+ A AA A A +-| > | A A A A A A AA A A | > 210 |-+ A A A A A A A-| > | A | > 200 |-+ A +-| > | | > 190 |-+ +-| > | | > 180 |-A +-| > | | > 170 |-+ +-| > | B B B | > 160 |-B B B B B B B +-| > | B B B B B | > 150 |-+ BB B BB B B B B B B B B-| > | B + B + + B + B B + B + B B + B | > 140 +-----------------------------------------------------------------------+ > 0 5 10 15 20 25 30 35 > 40 > iteration > 1.0.3-1 A > patched B > > > [0] see: https://lists.proxmox.com/pipermail/pbs-devel/2021-April/002932.html > > > proxmox-backup: Stefan Reiter (7): > tools/BroadcastFuture: add testcase for better understanding > tools: add AsyncLruCache as a wrapper around sync LruCache > backup: add CachedChunkReader utilizing AsyncLruCache > backup: add AsyncRead/Seek to CachedChunkReader > replace AsyncIndexReader with SeekableCachedChunkReader > backup: remove AsyncIndexReader > tools/lru_cache: make minimum capacity 1 > > src/api2/admin/datastore.rs | 4 +- > src/backup.rs | 4 +- > src/backup/async_index_reader.rs | 215 ------------------------- > src/backup/cached_chunk_reader.rs | 189 ++++++++++++++++++++++ > src/bin/proxmox_backup_client/mount.rs | 4 +- > src/tools.rs | 1 + > src/tools/async_lru_cache.rs | 135 ++++++++++++++++ > src/tools/broadcast_future.rs | 11 ++ > src/tools/lru_cache.rs | 1 + > 9 files changed, 343 insertions(+), 221 deletions(-) > delete mode 100644 src/backup/async_index_reader.rs > create mode 100644 src/backup/cached_chunk_reader.rs > create mode 100644 src/tools/async_lru_cache.rs > > proxmox-backup-qemu: Stefan Reiter (2): > add shared_cache module > access: use CachedChunkReader > > src/lib.rs | 7 ++++++- > src/restore.rs | 30 ++++++++---------------------- > src/shared_cache.rs | 37 +++++++++++++++++++++++++++++++++++++ > 3 files changed, 51 insertions(+), 23 deletions(-) > create mode 100644 src/shared_cache.rs > > -- > 2.30.2
_______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
