[pve-devel] [PATCH storage v6 3/7] api: content: support moving backups between path based storages

2025-01-20 Thread Filip Schauer
This commit adds the "backup+size" export format. When this format is used, the data stream starts with metadata of the backup (protected flag & notes) followed by the contents of the backup archive. Signed-off-by: Filip Schauer --- src/PVE/API2/Storage/Content.pm | 15 ++-- src/PVE/Stor

[pve-devel] [PATCH storage v6 7/7] storage migrate: avoid ssh when moving a volume locally

2025-01-20 Thread Filip Schauer
Avoid the overhead of SSH when moving a volume between storages on the same node. Signed-off-by: Filip Schauer --- src/PVE/Storage.pm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index 761f612..a2bef55 100755 --- a/src/PVE/Stor

[pve-devel] [PATCH storage v6 5/7] support moving VMA backups to PBS

2025-01-20 Thread Filip Schauer
Extend the move API to support moving VMA backups to a Proxmox Backup Server. Signed-off-by: Filip Schauer --- debian/control | 1 + src/PVE/API2/Storage/Content.pm | 53 +++ src/PVE/Storage/PBSPlugin.pm| 65 + 3 files

[pve-devel] [PATCH storage v6 0/7] support moving volumes between storages

2025-01-20 Thread Filip Schauer
Add the ability to move a backup, ISO, container template, snippet, or OVA/OVF between storages and nodes via an API method. Moving a VMA backup to a Proxmox Backup Server requires the proxmox-vma-to-pbs package to be installed. Currently only VMA backups can be moved to a Proxmox Backup Server and

[pve-devel] [PATCH storage v6 2/7] api: content: implement moving a volume between storages

2025-01-20 Thread Filip Schauer
Add the ability to move an iso, snippet or vztmpl between storages and nodes. Use either curl to call the API method: ``` curl https://$APINODE:8006/api2/json/nodes/$SOURCENODE/storage/$SOURCESTORAGE/content/$SOURCEVOLUME \ --insecure --cookie "$( --- src/PVE/API2/Storage/Content.pm | 111

[pve-devel] [PATCH storage v6 1/7] plugin: allow volume import of iso, snippets, vztmpl and import

2025-01-20 Thread Filip Schauer
Extend volume import functionality to support 'iso', 'snippets', 'vztmpl', and 'import' types, in addition to the existing support for 'images' and 'rootdir'. This is a prerequisite for the ability to move ISOs, snippets and container templates between nodes. Existing behavior for importing VM dis

[pve-devel] [PATCH storage v6 4/7] storage: introduce decompress_archive_into_pipe helper

2025-01-20 Thread Filip Schauer
Extract the file decompression code into its own reusable subroutine. Signed-off-by: Filip Schauer --- src/PVE/Storage.pm | 64 +- 1 file changed, 40 insertions(+), 24 deletions(-) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index 8e94979..76

[pve-devel] [PATCH storage v6 6/7] pvesm: add a move-volume command

2025-01-20 Thread Filip Schauer
The method can be called from the PVE shell with `pvesm move-volume`: ``` pvesm move-volume [--target-node ] [--delete] ``` For example to move a VMA backup to a Proxmox Backup Server: ``` pvesm move-volume \ local:backup/vzdump-qemu-100-2024_06_25-13_08_56.vma.zst pbs ``` Or move a cont

Re: [pve-devel] [PATCH storage v5 0/7] support moving volumes between storages

2025-01-20 Thread Filip Schauer
Merge conflicts to current master (e5f4af47d083) have been resolved in v6. Superseded by: https://lore.proxmox.com/pve-devel/20250120112842.36450-1-f.scha...@proxmox.com/ On 26/11/2024 16:23, Filip Schauer wrote: Add the ability to move a backup, ISO, container template, snippet, or OVA/OVF bet

[pve-devel] [PATCH qemu-server] api: create/update vm: avoid printing empty machine string

2025-01-20 Thread Fiona Ebner
While no problem is known with having an empty machine string in the configuration and it was already possible setting an empty machine manually via 'qm set', the behavior changed because of commit 919e69d0 ("machine: add check_and_pin_machine_string() helper") and there is potential for problemati

[pve-devel] applied-series: [PATCH-SERIES kernel] backport fix for kvm performance regression with Intel Emerald Rapids

2025-01-20 Thread Fiona Ebner
Am 19.12.24 um 15:29 schrieb Fiona Ebner: > Stumbled across this in the news [0]. > > Built, installed on a (CPU type 'host') VM and started a nested VM for > both 6.8 and 6.11 as a smoke test. Note I have an AMD TR 7960X, but > AFIACT, the patch affects a general code path not specific to Intel >

Re: [pve-devel] [PATCH v3 qemu-server 11/11] qcow2: add external snapshot support

2025-01-20 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- Hi Fabio ! >> >>In this implementation I don't see the possibility of using them on >>raw >>disks (on files) from a fast look, or am I wrong? If so, why? I think >>the main use would be in cases like that where you don't have >>snapshot >--- End Message ---

Re: [pve-devel] [PATCH v3 qemu-server 08/11] blockdev: convert drive_mirror to blockdev_mirror

2025-01-20 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- >>the path referenced in the running VM is stable. the path you are >>looking for in the graph is not. e.g., the path might be something >>some storage software returns. or udev. or .. and that can change >>with any software upgrade or not be 100% deterministic in the first >>

[pve-devel] applied: [PATCH qemu-server] api: create/update vm: avoid printing empty machine string

2025-01-20 Thread Fabian Grünbichler
On January 20, 2025 2:27 pm, Fiona Ebner wrote: > While no problem is known with having an empty machine string in the > configuration and it was already possible setting an empty machine > manually via 'qm set', the behavior changed because of commit 919e69d0 > ("machine: add check_and_pin_machine

[pve-devel] [PATCH qemu-server v5 05/11] migrate: call vm_stop_cleanup after stopping in phase3_cleanup

2025-01-20 Thread Dominik Csapak
we currently only call deactivate_volumes, but we actually want to call the whole vm_stop_cleanup, since that is not invoked by the vm_stop above (we cannot parse the config anymore) and might do other cleanups we also want to do (like mdev cleanup). For this to work properly we have to clone the

[pve-devel] [PATCH qemu-server v5 06/11] pci: set 'enable-migration' to on for live-migration marked mapped devices

2025-01-20 Thread Dominik Csapak
the default is 'auto', but for those which are marked as capable for live migration, we want to explicitly enable that, so we get an early error on start if the driver does not support that. Signed-off-by: Dominik Csapak --- PVE/QemuServer/PCI.pm | 9 - 1 file changed, 8 insertions(+), 1

[pve-devel] [PATCH guest-common/qemu-server/manager/docs v5 0/3] implement experimental vgpu live migration

2025-01-20 Thread Dominik Csapak
and some useful cleanups This is implemented for mapped resources. This requires driver and hardware support, but aside from nvidia vgpus there don't seem to be many drivers (if any) that do support that. qemu already supports that for vfio-pci devices, so nothing to be done there besides activel

[pve-devel] [PATCH qemu-server v5 02/11] pci: mapping: move implementation of find_on_current_node here

2025-01-20 Thread Dominik Csapak
this was the only user, and it's easy enough Signed-off-by: Dominik Csapak --- changes from v4: * factor out some code into it's own sub to keep parse_hostpci smaller (we need $config again later, so keep it outside the sub) PVE/QemuServer/PCI.pm | 15 +-- 1 file changed, 13 inser

[pve-devel] [PATCH manager v5 1/5] mapping: pci: include mdev in config checks

2025-01-20 Thread Dominik Csapak
by also providing the global config in assert_valid, and by also adding the mdev config in the 'toCheck' object in the gui For the gui, we extract the mdev property from the global entry, and add it to the individual mapping entries, that way we can reuse the checking logic of the other properties

[pve-devel] [PATCH guest-common v5 2/3] mapping: pci: add 'live-migration-capable' flag to mappings

2025-01-20 Thread Dominik Csapak
so that we can decide in qemu-server to allow live-migration. The driver and QEMU must be capable of that, and it's the admin's responsibility to know and configure that Mark the option as experimental in the description. Signed-off-by: Dominik Csapak --- src/PVE/Mapping/PCI.pm | 8 1

[pve-devel] [PATCH qemu 2/2] stable fixes for QEMU 9.2.0

2025-01-20 Thread Fiona Ebner
Most notabbly, there now is an upstream workaround for the "Windows PCI Label bug" [0] and the revert of QEMU commit 44d975ef34 ("x86: acpi: workaround Windows not handling name references in Package properly") can be dropped. Pick up some other fixes already merged in current master, for emulatio

[pve-devel] [PATCH-SERIES qemu 0/2] QEMU 9.2.0

2025-01-20 Thread Fiona Ebner
Very small diff in our patches this time and only a couple of stable fixes seemed worth picking up, see the patches for details. Basic tests for the following were done (disclaimer: not each combination of the mentioned settings): * OS boot testing: Win 11/10/Server 2019, Debian 12 (32 and 64 bit)

[pve-devel] [PATCH qemu-server] api: fix ova live import by using correct format for source image

2025-01-20 Thread Fiona Ebner
Commit c8ed1ac2 ("api: create disks: live import: use format from storage layer") broke live import from an OVA containing a disk, because a combined format like 'ova+vmdk' was used for the live-import disk mapping, leading to failure: > invalid format 'ova+vmdk' for 'scsi0' mapping There was als

Re: [pve-devel] [PATCH v3 qemu-server 11/11] qcow2: add external snapshot support

2025-01-20 Thread Fabio Fantoni via pve-devel
--- Begin Message --- Il 20/01/2025 14:44, DERUMIER, Alexandre ha scritto: Hi Fabio ! In this implementation I don't see the possibility of using them on raw disks (on files) from a fast look, or am I wrong? If so, why? I think the main use would be in cases like that where you don't have snaps

[pve-devel] [PATCH qemu-server v5 07/11] check_local_resources: add more info per mapped device and return as hash

2025-01-20 Thread Dominik Csapak
such as the mapping name and if it's marked for live-migration (pci only) Signed-off-by: Dominik Csapak --- PVE/API2/Qemu.pm | 2 +- PVE/QemuMigrate.pm | 7 --- PVE/QemuServer.pm | 17 ++--- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/PVE/API2/Qemu.pm b/

[pve-devel] [PATCH manager v5 4/5] ui: adapt migration window to precondition api change

2025-01-20 Thread Dominik Csapak
we now return the 'allowed_nodes'/'not_allowed_nodes' also if the vm is running, when it has mapped resources. So do that checks independently so that the user has instant feedback where those resources exist. Signed-off-by: Dominik Csapak --- www/manager6/window/Migrate.js | 26 +---

[pve-devel] [PATCH guest-common v5 3/3] mapping: remove find_on_current_node

2025-01-20 Thread Dominik Csapak
they only have one user each (where we can inline the implementation). It's easy enough to recreate should we need to. Signed-off-by: Dominik Csapak --- breaks older qemu-server that rely on this function src/PVE/Mapping/PCI.pm | 11 --- src/PVE/Mapping/USB.pm | 10 -- 2 files c

[pve-devel] [PATCH manager v5 3/5] bulk migrate: include checks for live-migratable local resources

2025-01-20 Thread Dominik Csapak
those should be able to migrate even for online vms. If the mapping does not exist on the target node, that will be caught further down anyway. Signed-off-by: Dominik Csapak --- PVE/API2/Nodes.pm | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/PVE/API2/Nodes.pm

[pve-devel] [PATCH docs v5 1/2] qm: resource mapping: add description for `mdev` option

2025-01-20 Thread Dominik Csapak
in a new section about additional options Signed-off-by: Dominik Csapak --- changes from v4: * use different asciidoc style to mark section qm.adoc | 12 1 file changed, 12 insertions(+) diff --git a/qm.adoc b/qm.adoc index 4bb8f2c..6f337fe 100644 --- a/qm.adoc +++ b/qm.adoc @@ -1

[pve-devel] [PATCH qemu-server v5 09/11] api: include not mapped resources for running vms in migrate preconditions

2025-01-20 Thread Dominik Csapak
so that we can show a proper warning in the migrate dialog and check it in the bulk migrate precondition check the unavailable_storages and should be the same as before, but we now always return (not_)allowed_nodes too. to make the code a bit easier to read, reorganize how we construct the (not_)

[pve-devel] [PATCH manager v5 5/5] fix #5175: ui: allow configuring and live migration of mapped pci resources

2025-01-20 Thread Dominik Csapak
if the hardware/driver is capable, the admin can now mark a pci device as 'live-migration-capable', which then tries enabling live migration for such devices. mark it as experimental when configuring and in the migrate window Signed-off-by: Dominik Csapak --- www/manager6/window/Migrate.js|

[pve-devel] [PATCH guest-common v5 1/3] mapping: pci: check the mdev configuration on the device too

2025-01-20 Thread Dominik Csapak
but that lives int he 'global' part of the mapping config, not in a specific mapping. To check that, add it to the $configured_props from there. this requires all call sites to be adapted otherwise the check will always fail for devices that are capable of mediated devices Signed-off-by: Dominik

[pve-devel] [PATCH qemu-server v5 03/11] pci: mapping: check mdev config against hardware

2025-01-20 Thread Dominik Csapak
by giving the mapping config to assert_valid, not only the specific mapping Signed-off-by: Dominik Csapak --- PVE/QemuServer/PCI.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PVE/QemuServer/PCI.pm b/PVE/QemuServer/PCI.pm index d6655b76..ad6d5ff6 100644 --- a/PVE/QemuServ

[pve-devel] [PATCH manager v5 2/5] bulk migrate: improve precondition checks

2025-01-20 Thread Dominik Csapak
this now takes into account the 'not_allowed_nodes' hash we get from the api call. With that, we can now limit the 'local_resources' check for online vms only, as for offline guests, the 'unavailable-resources' hash already includes mapped devices that don't exist on the target node. This now also

[pve-devel] [PATCH qemu-server v5 10/11] tests: cfg2cmd: fix mdev tests

2025-01-20 Thread Dominik Csapak
this will fail with the new checks for mdev when we don't have the correct config. namely a device that has mediated devices, should have 'mdev' set in the mapping config Signed-off-by: Dominik Csapak --- test/run_config2command_tests.pl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) di

[pve-devel] [PATCH qemu-server v5 08/11] api: enable live migration for marked mapped pci devices

2025-01-20 Thread Dominik Csapak
They have to be marked as 'live-migration-capable' in the mapping config, and the driver and qemu must support it. For the gui checks, we now return the whole object of the mapped resources, which includes info like the name and if it's marked as live-migration capable. (while deprecating the old

[pve-devel] [PATCH qemu-server v5 04/11] vm stop-cleanup: allow callers to decide error behavior

2025-01-20 Thread Dominik Csapak
and keep it the same for all current callers as before by setting the additional 'noerr' parameter to '1'. Signed-off-by: Dominik Csapak --- PVE/CLI/qm.pm | 2 +- PVE/QemuServer.pm | 13 - 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.

[pve-devel] [PATCH qemu-server v5 11/11] migration: show vfio state transferred too

2025-01-20 Thread Dominik Csapak
there is no total here, so we can't show any, just what was transferred up until now (if any). Signed-off-by: Dominik Csapak --- PVE/API2/Qemu.pm | 2 +- PVE/QemuMigrate.pm | 12 +++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm i

[pve-devel] [PATCH docs v5 2/2] qm: resource mapping: document `live-migration-capable` setting

2025-01-20 Thread Dominik Csapak
Signed-off-by: Dominik Csapak --- changes from v4: * use same wording as the previous option (`the PCI device`) qm.adoc | 6 ++ 1 file changed, 6 insertions(+) diff --git a/qm.adoc b/qm.adoc index 6f337fe..0d18d7e 100644 --- a/qm.adoc +++ b/qm.adoc @@ -1955,6 +1955,12 @@ Currently there are

Re: [pve-devel] [PATCH v3 qemu-server 11/11] qcow2: add external snapshot support

2025-01-20 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- >>out of curiosity, besides the obvious cases where external snapshots >>would be useful, i.e. on raw files and on lvm (not lvm-thin), what >>other >>cases would be useful given that they already have snapshot support >>(qcow2 with the internal one, zfs, btrfs, lvm-thin etc

[pve-devel] [PATCH pve-kernel] buildsys: add explicit DEB_VERSION variable in Makefile

2025-01-20 Thread Stoiko Ivanov
Can be used to generate a package for testing a particular bugfix, or new version, which can be shared with a small audience, without reusing the current package version, or bumping to a version that looks like our released kernels[0]. previously this could be accomplished by setting the PKGREL va

[pve-devel] [PATCH qemu-server v5 01/11] usb: mapping: move implementation of find_on_current_node here

2025-01-20 Thread Dominik Csapak
this was the only user, and it's easy enough Signed-off-by: Dominik Csapak --- changes from v4: * refactor code into sub to keep the 'parse_usb_device' function smaller PVE/QemuServer/USB.pm | 17 ++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/PVE/QemuServer/US

[pve-devel] [PATCH docs] qm, vzdump: add notes regarding backups of windows and VSS

2025-01-20 Thread Aaron Lauterer
VSS in windows guests with additional backup software in the guest can cause problems. By now the guest agent does support to use a different VSS option. This way, one can switch to one that does not interfere with other backup solutions. Mention it in the docs and link to the wiki article that ex

[pve-devel] applied: [PATCH qemu-server] api: fix ova live import by using correct format for source image

2025-01-20 Thread Thomas Lamprecht
Am 20.01.25 um 16:57 schrieb Fiona Ebner: > Commit c8ed1ac2 ("api: create disks: live import: use format from > storage layer") broke live import from an OVA containing a disk, > because a combined format like 'ova+vmdk' was used for the live-import > disk mapping, leading to failure: > >> invalid

[pve-devel] [PATCH qemu 1/2] update submodule and patches to QEMU 9.2.0

2025-01-20 Thread Fiona Ebner
Notable changes: * Commit 07bea2d35f ("block-backend: Remove deadcode") removed blk_op_{,un}block_all() which was used by PVE async savevm code. Fixed by switching to using bdrv_op_{,un}block_all(). * Drop patches that are already part of upstream. Signed-off-by: Fiona Ebner --- ...d-suppo