[pve-devel] [PATCH docs v2 21/21] vzdump: add section about backup fleecing

2024-03-15 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- Changes in v2: * flesh out recommendations much more * adapt to changes in backend (i.e. using qcow2 if available, no storage default) vzdump.adoc | 38 ++ 1 file changed, 38 insertions(+) diff --git a/vzdump.adoc

[pve-devel] [PATCH manager v2 13/21] api: backup/vzdump: add permission check for fleecing storage

2024-03-15 Thread Fiona Ebner
Similar to how Datastore.AllocateSpace is required for the backup storage, it should also be required for the fleecing storage. Removing a fleecing storage from a job does not require more permissions than for modifying the job. Suggested-by: Fabian Grünbichler Signed-off-by: Fiona Ebner --- N

[pve-devel] [RFC qemu-server v2 18/21] vzdump: better cleanup fleecing images after hard errors

2024-03-15 Thread Fiona Ebner
By recording the allocated fleecing images in the VM config, they are not immediately orphaned, should a hard error occur during backup that prevents cleanup. They are attempted to be cleaned up during the next backup run. Suggested-by: Fabian Grünbichler Signed-off-by: Fiona Ebner --- New in

[pve-devel] [PATCH qemu v2 07/21] PVE backup: add fleecing option

2024-03-15 Thread Fiona Ebner
When a fleecing option is given, it is expected that each device has a corresponding "-fleecing" block device already attached, except for EFI disk and TPM state, where fleecing is never used. The following graph was adapted from [0] which also contains more details about fleecing. [guest] |

[pve-devel] [PATCH qemu-server v2 15/21] backup: implement fleecing option

2024-03-15 Thread Fiona Ebner
Management for fleecing images is implemented here. If the fleecing option is set, for each disk (except EFI disk and TPM state) a new fleecing image is allocated on the configured fleecing storage (same storage as original disk by default). The disk is attached to QEMU with the 'size' parameter, b

[pve-devel] [PATCH qemu v2 04/21] qapi: blockdev-backup: add discard-source parameter

2024-03-15 Thread Fiona Ebner
From: Vladimir Sementsov-Ogievskiy Add a parameter that enables discard-after-copy. That is mostly useful in "push backup with fleecing" scheme, when source is snapshot-access format driver node, based on copy-before-write filter snapshot-access API: [guest] [snapshot-access] ~~ blockdev-ba

[pve-devel] [PATCH qemu v2 03/21] block/copy-before-write: create block_copy bitmap in filter node

2024-03-15 Thread Fiona Ebner
From: Vladimir Sementsov-Ogievskiy Currently block_copy creates copy_bitmap in source node. But that is in bad relation with .independent_close=true of copy-before-write filter: source node may be detached and removed before .bdrv_close() handler called, which should call block_copy_state_free(),

[pve-devel] [RFC qemu-server v2 17/21] schema: add fleecing-images config property

2024-03-15 Thread Fiona Ebner
to be used internally to record volume IDs of fleecing images allocated during backup. Suggested-by: Fabian Grünbichler Signed-off-by: Fiona Ebner --- New in v2. PVE/API2/Qemu.pm | 9 + PVE/QemuServer.pm| 7 +++ PVE/VZDump/QemuServer.pm | 1 + 3 files changed, 17 i

[pve-devel] [PATCH qemu v2 05/21] copy-before-write: allow specifying minimum cluster size

2024-03-15 Thread Fiona Ebner
Useful to make discard-source work in the context of backup fleecing when the fleecing image has a larger granularity than the backup target. Copy-before-write operations will use at least this granularity and in particular, discard requests to the source node will too. If the granularity is too s

[pve-devel] [RFC qemu-server v2 19/21] migration: attempt to clean up potential left-over fleecing images

2024-03-15 Thread Fiona Ebner
Before the guest is migrated to a different node and they'd really become orphaned. Suggested-by: Fabian Grünbichler Signed-off-by: Fiona Ebner --- New in v2. PVE/QemuMigrate.pm | 3 +++ 1 file changed, 3 insertions(+) diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm index 8d9b35ae..a190

[pve-devel] [PATCH qemu v2 06/21] backup: add minimum cluster size to performance options

2024-03-15 Thread Fiona Ebner
Useful to make discard-source work in the context of backup fleecing when the fleecing image has a larger granularity than the backup target. Backup/block-copy will use at least this granularity for copy operations and in particular, discard requests to the backup source will too. If the granulari

[pve-devel] [RFC guest-common v2 11/21] abstract config: do not copy fleecing images entry for snapshot

2024-03-15 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- RFC because belongs to the optional "improved cleanup"-part of the series. See qemu-server patches for more info. New in v2. src/PVE/AbstractConfig.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/src/PVE/AbstractConfig.pm b/src/PVE/AbstractConfig.pm index

[pve-devel] [PATCH common v2 08/21] json schema: add format description for pve-storage-id standard option

2024-03-15 Thread Fiona Ebner
so that the option can be used as part of a property string. Signed-off-by: Fiona Ebner --- New in v2. src/PVE/JSONSchema.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/src/PVE/JSONSchema.pm b/src/PVE/JSONSchema.pm index 7be0595..a74bbe2 100644 --- a/src/PVE/JSONSchema.pm +++ b/src/PVE

[pve-devel] [PATCH qemu v2 01/21] block/copy-before-write: fix permission

2024-03-15 Thread Fiona Ebner
From: Vladimir Sementsov-Ogievskiy In case when source node does not have any parents, the condition still works as required: backup job do create the parent by block_job_create -> block_job_add_bdrv -> bdrv_root_attach_child Still, in this case checking @perm variable doesn't work, as backup

[pve-devel] [PATCH guest-common v2 09/21] vzdump: schema: add fleecing property string

2024-03-15 Thread Fiona Ebner
It's a property string, because that avoids having an implicit "enabled" as part of a 'fleecing-storage' property. And there likely will be more options in the future, e.g. threshold/limit for the fleecing image size. Storage is non-optional, so the storage choice needs to be a conscious decision.

[pve-devel] [RFC qemu-server v2 20/21] destroy vm: clean up potential left-over fleecing images

2024-03-15 Thread Fiona Ebner
To avoid them becoming orphaned. Suggested-by: Fabian Grünbichler Signed-off-by: Fiona Ebner --- New in v2. PVE/QemuServer.pm | 3 +++ 1 file changed, 3 insertions(+) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index 91e34166..b80286f3 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServ

[pve-devel] [RFC qemu-server v2 16/21] parse config: allow config keys with minus sign

2024-03-15 Thread Fiona Ebner
In preparation for the upcoming 'fleecing-images' key. To avoid mixing of options with - and options with _, which is not very user-friendly, it would be nice to add aliases for existing options with _. And long-term, backup restore handlers could switch to the modern keys with -. Signed-off-by: F

[pve-devel] [PATCH qemu-server v2 14/21] backup: disk info: also keep track of size

2024-03-15 Thread Fiona Ebner
which will be needed to allocate fleecing images. Signed-off-by: Fiona Ebner --- No changes in v2. PVE/VZDump/QemuServer.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/PVE/VZDump/QemuServer.pm b/PVE/VZDump/QemuServer.pm index be7d8e1e..51498dbc 100644 --- a/PVE/VZDump/QemuServer.pm +++

[pve-devel] [PATCH manager v2 12/21] vzdump: handle new 'fleecing' property string

2024-03-15 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- No changes in v2. PVE/VZDump.pm | 12 1 file changed, 12 insertions(+) diff --git a/PVE/VZDump.pm b/PVE/VZDump.pm index 152eb3e5..74eb0c83 100644 --- a/PVE/VZDump.pm +++ b/PVE/VZDump.pm @@ -130,6 +130,15 @@ my $generate_notes = sub { return $not

[pve-devel] [PATCH guest-common v2 10/21] vzdump: schema: make storage for fleecing semi-optional

2024-03-15 Thread Fiona Ebner
so it doesn't need to be set when explicitly disabling fleecing. Needs a custom verifier to enforce it being set when enabled. Suggested-by: Fabian Grünbichler Signed-off-by: Fiona Ebner --- New in v2. src/PVE/VZDump/Common.pm | 14 +- 1 file changed, 13 insertions(+), 1 deletion(

[pve-devel] [PATCH installer] build: run shellcheck as part of `test` step

2024-03-15 Thread Christoph Heiss
Especially unconfigured.sh is worth checking consistently. Running shellcheck also does not really have any notable impact on build time, so no downside there either. Signed-off-by: Christoph Heiss --- Makefile| 14 +- debian/control | 1 + unconfigured.sh | 3 +++ xinitr

[pve-devel] [PATCH-SERIES v2] fix #4136: implement backup fleecing

2024-03-15 Thread Fiona Ebner
Changes in v2 (thanks - not limited to - to Fabian and Alexandre for feedback!): * Use v3 of "discard-source" upstream series (v4 was posted in the meantime but without any semantic change) * Add patches to specify minimum cluster size during backup, to allow discard to work eve

[pve-devel] [PATCH qemu v2 02/21] block/copy-before-write: support unligned snapshot-discard

2024-03-15 Thread Fiona Ebner
From: Vladimir Sementsov-Ogievskiy First thing that crashes on unligned access here is bdrv_reset_dirty_bitmap(). Correct way is to align-down the snapshot-discard request. Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Fiona Ebner --- Changes in v2: * update to latest upstrea

Re: [pve-devel] [PATCH storage 1/1] storage/plugins: pass scfg to parse_volname

2024-03-15 Thread Roland Kammerer via pve-devel
--- Begin Message --- On Wed, Mar 13, 2024 at 04:38:57PM +0100, Wolfgang Bumiller wrote: > My thoughts on this: (TLDR: we should just merge it and probably also > consider adding a separate method to get the *format* of a volid) > > - Adding the parameter itself is fine, not thinking about how/why