Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

2024-07-29 Thread Jonathan Nicklin via pve-devel
work going on in this space :) Between your and Dietmar's replies, I see the constraints and potential avenues for improvement. Thanks for your reply! Respectfully, -Jonathan > On Jul 29, 2024, at 4:15 AM, Fiona Ebner wrote: > > Hi, > > Am 26.07.24 um 21:47 schrieb Jona

Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

2024-07-29 Thread Jonathan Nicklin via pve-devel
--- Begin Message --- I am by no means a CEPH expert. However, my understanding is that other backup solutions (in the OpenStack world) have used rbd diff to enable incremental backups. I was hoping that would be relevant here. Here's the description of `rbd diff` Dump a list of byte extents

Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

2024-07-29 Thread Jonathan Nicklin via pve-devel
--- Begin Message --- In hyper-converged deployments, the node performing the backup is sourcing ((nodes-1)/(nodes))*bytes) of backup data (i.e., ingress traffic) and then sending 1*bytes to PBS (i.e., egress traffic). If PBS were to pull the data from the nodes directly, the maximum load on any

Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

2024-07-29 Thread Jonathan Nicklin via pve-devel
--- Begin Message --- > On Jul 27, 2024, at 11:20 AM, Dietmar Maurer wrote: > >> Would adding support for offloading incremental difference detection >> to the underlying storage be feasible with the API updates? The QEMU >> bitmap strategy works for all storage devices but is far from >> optima

Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

2024-07-27 Thread Jonathan Nicklin via pve-devel
--- Begin Message --- Hi Fiona, Would adding support for offloading incremental difference detection to the underlying storage be feasible with the API updates? The QEMU bitmap strategy works for all storage devices but is far from optimal. If backup coordinated a storage snapshot, the underlying