On 16/03/2021 21:17, Thomas Lamprecht wrote:
On 03.03.21 10:56, Stefan Reiter wrote:
Values chosen by fair dice roll, seems to be a good sweet spot on my
machine where any less causes performance degradation but any more
doesn't really make it go any faster.
Keep in mind that those values are per drive in an actual restore.
Signed-off-by: Stefan Reiter <s.rei...@proxmox.com>
---
Depends on new proxmox-backup.
v2:
* unchanged
src/restore.rs | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/src/restore.rs b/src/restore.rs
index 0790d7f..a1acce4 100644
--- a/src/restore.rs
+++ b/src/restore.rs
@@ -218,15 +218,16 @@ impl RestoreTask {
let index = client.download_fixed_index(&manifest, &archive_name).await?;
let archive_size = index.index_bytes();
- let most_used = index.find_most_used_chunks(8);
+ let most_used = index.find_most_used_chunks(16); // 64 MB most used
cache
let file_info = manifest.lookup_file_info(&archive_name)?;
- let chunk_reader = RemoteChunkReader::new(
+ let chunk_reader = RemoteChunkReader::new_lru_cached(
Arc::clone(&client),
self.crypt_config.clone(),
file_info.chunk_crypt_mode(),
most_used,
+ 64, // 256 MB LRU cache
how does this work with low(er) memory situations? Lots of people do not over
dimension their memory that much, and especially the need for mass-recovery
could
seem to correlate with reduced resource availability (a node failed, now I need
to restore X backups on my <test/old/other-already-in-use> node, so multiple
restore jobs may run in parallel, and they all may have even multiple disks,
so tens of GiB of memory just for the cache are not that unlikely.
This is a seperate function from the regular restore, so it currently
only affects live-restore. This is not an operation you would usually do
under memory constraints anyway, and regular restore is unaffected if
you just want the data.
Upcoming single-file restore too though, I suppose, where it might make
more sense...
How is the behavior, hard failure if memory is not available? Also, some
archives
may be smaller than 256 MiB (EFI disk??) so there it'd be weird to have 256
cache
and get 64 of most used chunks if that's all/more than it would actually need to
be..
Yes, if memory is unavailable it is a hard error. Memory should not be
pre-allocated however, so restoring this way will only ever use as much
memory as the disk size (not accounting for overhead).
There may be the reversed situation too, beefy fast node with lots of memory
and restore is used as recovery or migration but network bw/latency to PBS is
not
that good - so bigger cache could be wanted.
The reason I chose the numbers I did was that I couldn't see any real
performance benefits by going higher, though I didn't specifically test
with slow networking.
I don't believe more cache would improve the situation there though,
this is mostly to avoid random access from the guest and the linear
access from the block-stream operation to interfere with each other, and
allow multiple smaller guest reads within the same chunk to be served
quickly.
Maybe we could get the available memory and use that as hint, I mean as memory
usage can be highly dynamic it will never be perfect, but better than just
ignoring
it..
If anything, I'd make it user-configurable - I don't think a heuristic
would be a good choice here.
This way we could also set it smaller for single-file restore for
example - on the other hand, that adds another parameter to the already
somewhat cluttered QEMU<->Rust interface.
);
let reader = AsyncIndexReader::new(index, chunk_reader);
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
