Re: [pve-devel] [PATCH storage 1/2] fix #5779: rbd: allow to pass custom krbd map options

2024-12-19 Thread Friedrich Weber
On 30/10/2024 17:49, Friedrich Weber wrote: > [...] > > Yeah, I see the point. > > Of course, another alternative is enabling `rxbounce` unconditionally, > as initially requested in [1]. I'm a hesitant to do that because from > reading its description I'd expect it could have a performance impact

Re: [pve-devel] [PATCH v2 storage 05/10] rbd plugin: implement volume import/export

2024-12-19 Thread Fiona Ebner
Am 18.12.24 um 16:33 schrieb DERUMIER, Alexandre via pve-devel: >>> Am 18.12.24 um 15:20 schrieb Daniel Kral: - When exporting with "pvesm export ...", the volume has the same >> checksum as with "rbd export ..." with the size header prepended > >>> Well, I totally missed the existence of "rb

[pve-devel] [PATCH storage v3 01/13] plugin: export/import: fix calls to path() method

2024-12-19 Thread Fiona Ebner
The plugin API does not require call context detection for the returned value of the path() method. See other plugins like ISCSIDirect/ZFS that do not implement it. So do not expect it for external plugins either. Signed-off-by: Fiona Ebner --- New in v3. src/PVE/Storage/Plugin.pm | 8

[pve-devel] [PATCH storage v3 06/13] rbd plugin: improve volume exists helper

2024-12-19 Thread Fiona Ebner
Currently, the helper would not distinguish between different kinds of errors. Instead of relying on an error, list the images and check there. Signed-off-by: Fiona Ebner --- Changes in v3: * do not match error message, list images instead src/PVE/Storage/RBDPlugin.pm | 23 ++--

[pve-devel] [PATCH storage v3 09/13] common: introduce common module

2024-12-19 Thread Fiona Ebner
From: Max Carrara This module's purpose is to provide shared functions, constants, etc. for storage plugins and storage-related operations. It starts out with a align_size_up() function, that will (initially) be used for volume import. Originally-by: Max Carrara [FE: start out with a different

[pve-devel] [PATCH storage v3 11/13] rbd plugin: list: drop outdated error message check

2024-12-19 Thread Fiona Ebner
This became outdated after Ceph commit ac547a5b7dc ("rbd: return 0 and an empty list when pool is entirely empty") 11 years ago. See also: https://tracker.ceph.com/issues/6693 Signed-off-by: Fiona Ebner --- New in v3. src/PVE/Storage/RBDPlugin.pm | 7 +-- 1 file changed, 1 insertion(+), 6

[pve-devel] [PATCH container v3 12/13] migration: allow rbd storages for remote migration

2024-12-19 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- src/PVE/LXC/Migrate.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/src/PVE/LXC/Migrate.pm b/src/PVE/LXC/Migrate.pm index 0debb9e..e1e6cab 100644 --- a/src/PVE/LXC/Migrate.pm +++ b/src/PVE/LXC/Migrate.pm @@ -303,6 +303,7 @@ sub phase1 { if ($remot

[pve-devel] [PATCH storage v3 05/13] rbd plugin: implement volume import/export

2024-12-19 Thread Fiona Ebner
For now, only 'raw+size' is supported and it's not possible to export/import with snapshots. The volume or snapshot is exported or imported via the corresponding 'rbd' commands. Introducing an 'rbd' transport format might be feasible for more complete (i.e. with snapshots, incremental) transfer be

[pve-devel] [PATCH storage v3 08/13] iscsi direct plugin: support volume export

2024-12-19 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- src/PVE/Storage/ISCSIDirectPlugin.pm | 63 1 file changed, 63 insertions(+) diff --git a/src/PVE/Storage/ISCSIDirectPlugin.pm b/src/PVE/Storage/ISCSIDirectPlugin.pm index eb329d4..0aca4b6 100644 --- a/src/PVE/Storage/ISCSIDirectPlugin.

[pve-devel] [PATCH storage/container v3 00/13] import/export for shared storages

2024-12-19 Thread Fiona Ebner
Chnages in v3: * add fix for path() calls in Plugin.pm * use rbd export/import instead of mapping+dd * list images for volume exists helper * fix commit message for patch adding Common module * add patch dropping outdated error message check Changes in v2: * add fix for path() in iSCSI direct plug

[pve-devel] [PATCH storage v3 02/13] rbd plugin: schema: document default value for 'krbd' setting

2024-12-19 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- src/PVE/Storage/RBDPlugin.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm index f45ad3f..680e922 100644 --- a/src/PVE/Storage/RBDPlugin.pm +++ b/src/PVE/Storage/RBDPlugin.pm @@ -389,6 +389,7 @@ sub p

[pve-devel] [PATCH storage v3 03/13] export: redirect stdout to avoid any unrelated messages ending up in the export stream

2024-12-19 Thread Fiona Ebner
Current export implementations luckily seems to not run into this issue yet. However, for the upcoming implementation for RBD, mapping a volume would print the device path to STDOUT, thus messing up the export stream. Signed-off-by: Fiona Ebner --- src/PVE/CLI/pvesm.pm | 5 - 1 file changed,

[pve-devel] [PATCH storage v3 07/13] iscsi plugin: support volume export

2024-12-19 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- src/PVE/Storage/ISCSIPlugin.pm | 48 ++ 1 file changed, 48 insertions(+) diff --git a/src/PVE/Storage/ISCSIPlugin.pm b/src/PVE/Storage/ISCSIPlugin.pm index 6de7610..eb70453 100644 --- a/src/PVE/Storage/ISCSIPlugin.pm +++ b/src/PVE/St

[pve-devel] [PATCH storage v3 04/13] rbd plugin: factor out helper to check if volume already exists

2024-12-19 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- src/PVE/Storage/RBDPlugin.pm | 17 - 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm index 680e922..301918c 100644 --- a/src/PVE/Storage/RBDPlugin.pm +++ b/src/PVE/Storage/RBD

[pve-devel] [PATCH container v3 13/13] migration: add reminder to evaluate dropping seemingly useless check for PVE 9

2024-12-19 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- src/PVE/LXC/Migrate.pm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/PVE/LXC/Migrate.pm b/src/PVE/LXC/Migrate.pm index e1e6cab..a550d70 100644 --- a/src/PVE/LXC/Migrate.pm +++ b/src/PVE/LXC/Migrate.pm @@ -292,6 +292,8 @@ sub phase1 { my ($sid

Re: [pve-devel] [PATCH v2 storage 05/10] rbd plugin: implement volume import/export

2024-12-19 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- > No related, but could it be possible to implement it, for simple > vm/template full cloning with source+target are both rbd ? It's > really > faster with 'qemu-img convert' >>Hmm, we could shift offline copy of images to the storage layer (at >>least in some cases). We just

[pve-devel] [PATCH storage v3 10/13] plugins: volume import: align size up to 1KiB

2024-12-19 Thread Fiona Ebner
Previously, the size was rounded down which, in case of an image with non-1KiB-aligned sze (only possible for external plugins or manually created images) would lead to errors when attempting to write beyond the end of the too small allocated target image. For image allocation, the size is already

Re: [pve-devel] [PATCH storage v3 08/13] iscsi direct plugin: support volume export

2024-12-19 Thread Fiona Ebner
Am 19.12.24 um 11:43 schrieb Fiona Ebner: > Signed-off-by: Fiona Ebner Sorry, forgot to mention the changes here: * do not rely on path() using wantarray ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/

Re: [pve-devel] [PATCH v2 container 2/7] status: add pool usage fields

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > these are similar to existing ones, but with slightly different semantics. > > Signed-off-by: Fabian Grünbichler > --- > src/PVE/LXC.pm | 29 + > 1 file changed, 29 insertions(+) > > diff --git a/src/PVE/LXC.pm b/src

Re: [pve-devel] [PATCH v2 container 5/7] hotplug: handle pool limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > by checking the new values against the running limits. > > Signed-off-by: Fabian Grünbichler > --- > src/PVE/LXC/Config.pm | 13 + > 1 file changed, 13 insertions(+) > > diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm > i

Re: [pve-devel] [PATCH v2 guest-common 1/1] helpers: add pool limit/usage helpers

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > one for combining the per-node broadcasted values, one for checking a pool's > limit, and one specific helper for checking guest-related actions such as > starting a VM. > > Signed-off-by: Fabian Grünbichler > --- > > Notes: > v2: > - st

Re: [pve-devel] [PATCH v2 container 7/7] update: handle pool limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > Signed-off-by: Fabian Grünbichler > --- > > Notes: > v2: > - don't multiply mem+swap, but add them up (thanks Dominik) > > src/PVE/API2/LXC/Config.pm | 21 + > 1 file changed, 21 insertions(+) > > diff --git a/src/P

Re: [pve-devel] [PATCH v2 manager 4/4] ui: add pool limits and usage

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > Signed-off-by: Fabian Grünbichler > --- > > Notes: > this is very "bare", obviously we'd want > - a nicer grid/.. display of usage > - a way to edit the limits > > I am not yet sure how to integrate this nicely, and wanted to

Re: [pve-devel] [PATCH v2 qemu-server 1/6] config: add pool usage helper

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > determining the usage values for the current config. pending values are taken > into account if they are higher than the current value only, else it would be > possible to easily circumvent config limits by setting non-hotpluggable > pending > valu

Re: [pve-devel] [PATCH v2 qemu-server 4/6] update/hotplug: handle pool limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > if the new value is higher than the old one, check against limits. if the old > one is higher, then the change is always okay, to support reducing the usage > in > steps spread over multiple guests.. > > Signed-off-by: Fabian Grünbichler > --- >

Re: [pve-devel] [PATCH v2 qemu-server 5/6] start: handle pool limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > if the start is not part of an incoming migration, check the VM against its > pool's run limit. > > Signed-off-by: Fabian Grünbichler > --- > PVE/QemuServer.pm | 13 + > 1 file changed, 13 insertions(+) > > diff --git a/PVE/QemuServ

Re: [pve-devel] [PATCH v2 manager 1/4] api: pools: add limits management

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > allow to set/update limits, and return them when querying individual pools. > > Signed-off-by: Fabian Grünbichler > --- > > Notes: > requires bumped pve-access-control > > v2: > - unify run vs config limit checks into helper >

Re: [pve-devel] [PATCH v2 manager 2/4] pvestatd: collect and broadcast pool usage

2024-12-19 Thread Daniel Kral
This could only be partially applied because of changes to `update_{qemu,lxc}_status` in the mean time: Patch failed at 0002 pvestatd: collect and broadcast pool usage patching file PVE/Service/pvestatd.pm Hunk #1 FAILED at 231. Hunk #2 succeeded at 245 (offset 3 lines). Hunk #3 succeeded at 283 w

Re: [pve-devel] [PATCH v2 qemu-server 2/6] vmstatus: add usage values for pool limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > these are separate from the existing ones to allow changes on either end > without side-effects, since the semantics are not quite the same. > > the conf values incorporate pending values (if higher than the current config > value), and avoid clamp

[pve-devel] [PATCH pve-network] controllers: bgp: split v4 && v6 peers in different groups

2024-12-19 Thread Alexandre Derumier via pve-devel
--- Begin Message --- reported by user on the forum: https://forum.proxmox.com/threads/sdn-mismatch-afi-with-bgp-controller-ipv6-session.159250/ This is for dualstack, when evpn is ipv4, and bgp is ipv6+(ipv4) Signed-off-by: Alexandre Derumier --- src/PVE/Network/SDN/Controllers/BgpPlugin.pm |

[pve-devel] [PATCH pve-storage] qcow2: resize: add preallocation support

2024-12-19 Thread Alexandre Derumier via pve-devel
--- Begin Message --- Seem that we totally forgot to add it, it's available since 2017 https://www.mail-archive.com/qemu-devel@nongnu.org/msg436979.html --- src/PVE/Storage/Plugin.pm | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/

[pve-devel] [PATCH bookworm-6.8 kernel 3/4] patches: kernel: switch to using full index for patch files

2024-12-19 Thread Fiona Ebner
git will automatically change the length of the index hashes when formatting a patch depending on what references are present in the submodule. After pulling in the stable tags today, git wanted to add a character to all hashes for me. Use --full-index when generating the patches to avoid such issu

[pve-devel] [PATCH master kernel 1/4] patches: kernel: switch to using full index for patch files

2024-12-19 Thread Fiona Ebner
git will automatically change the length of the index hashes when formatting a patch depending on what references are present in the submodule. After pulling in the stable tags today, git wanted to add a character to all hashes for me. Use --full-index when generating the patches to avoid such issu

[pve-devel] [PATCH master kernel 2/4] backport fix for kvm performance regression with Intel Emerald Rapids

2024-12-19 Thread Fiona Ebner
Adapted to context change in "arch/x86/kvm/cpuid.h", because of the vcpu_supports_xsave_pkru() function that got added by Proxmox VE downstream patch "kvm: xsave set: mask-out PKRU bit in xfeatures if vCPU has no support". But otherwise clean cherry-pick from linux-next, no functional changes. Sig

[pve-devel] [PATCH-SERIES kernel] backport fix for kvm performance regression with Intel Emerald Rapids

2024-12-19 Thread Fiona Ebner
Stumbled across this in the news [0]. Built, installed on a (CPU type 'host') VM and started a nested VM for both 6.8 and 6.11 as a smoke test. Note I have an AMD TR 7960X, but AFIACT, the patch affects a general code path not specific to Intel CPUs. [0]: https://www.phoronix.com/news/Linux-6.13-

[pve-devel] [PATCH bookworm-6.8 kernel 4/4] backport fix for kvm performance regression with Intel Emerald Rapids

2024-12-19 Thread Fiona Ebner
Adapted to context change in "arch/x86/kvm/cpuid.h", because of the vcpu_supports_xsave_pkru() function that got added by Proxmox VE downstream patch "kvm: xsave set: mask-out PKRU bit in xfeatures if vCPU has no support". But otherwise clean cherry-pick from linux-next, no functional changes. Sig

[pve-devel] partially-applied: [PATCH storage/container v3 00/13] import/export for shared storages

2024-12-19 Thread Fabian Grünbichler
Folded in Filip's T-B from v2, and applied all but the last pve-container patch - those checks are there to cancel the migration early on if a storage that is not supported is contained in the mix.. ideally they should move to the storage layer, as the original comment indicates.. basically a help

Re: [pve-devel] [PATCH v2 qemu-server/pve-container 0/19] pool resource limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > high level description: > > VM/CT vmstatus returns new fields for configured and running "usage" > values, these are then broadcasted by pvestatd on each node via KV. > > helpers in guest-common to check those limits > pool API returns limits and

Re: [pve-devel] [PATCH v2 access-control 1/1] pools: define resource limits

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > and handle them when parsing/writing user.cfg > > Signed-off-by: Fabian Grünbichler > --- > > Notes: > - make limit schema public for pve-guest-common usage > > src/PVE/AccessControl.pm | 42 +-- > src/

Re: [pve-devel] [PATCH v2 container 1/7] config: add pool usage helper

2024-12-19 Thread Daniel Kral
On 16/04/2024 14:20, Fabian Grünbichler wrote: > to avoid repeating those calculations all over the place. > > Signed-off-by: Fabian Grünbichler > --- > src/PVE/LXC/Config.pm | 35 +++ > 1 file changed, 35 insertions(+) > > diff --git a/src/PVE/LXC/Config.pm b/

Re: [pve-devel] [ZFS on swap correction 1/1] Section on ZFS and swap corrected.

2024-12-19 Thread Fiona Ebner
We already discussed this off-list yesterday, but mentioning it here for completeness and for the mail archive: The subject prefix for the title should be "PATCH docs" or "PATCH pve-docs". E.g., this can be configured in .git/config for the repository > [format] > subjectprefix = PATCH docs

Re: [pve-devel] [PATCH master kernel 1/4] patches: kernel: switch to using full index for patch files

2024-12-19 Thread Fiona Ebner
Am 19.12.24 um 15:29 schrieb Fiona Ebner: > git will automatically change the length of the index hashes when > formatting a patch depending on what references are present in the > submodule. After pulling in the stable tags today, git wanted to add > a character to all hashes for me. Use --full-in

Re: [pve-devel] partially-applied: [PATCH storage/container v3 00/13] import/export for shared storages

2024-12-19 Thread Fiona Ebner
Am 19.12.24 um 12:45 schrieb Fabian Grünbichler: > Folded in Filip's T-B from v2, and applied all but the last > pve-container patch - those checks are there to cancel the migration > early on if a storage that is not supported is contained in the mix.. > > ideally they should move to the storage

Re: [pve-devel] [Typo corrected. 2/2] Split a long sentence and corrected some typos.

2024-12-19 Thread Fiona Ebner
If you send another version of a patch series, please use the "-v , --reroll-count=" option for "git format-patch". As long as you are working on a series, you can "change history" on your local branch. This can be done with "git rebase -i" and "git commit --amend" and (there are also things like

[pve-devel] [PATCH v1 pve-common 3/4] introduce PVE::Filesystem

2024-12-19 Thread Max Carrara
The PVE::Filesystem module implements filesystem manipulation operations. Currently, this is limited to two wrapper functions, but in the future anything that concernts itself with altering the filesystem should be added to it. This module can be seen as the "complement" to PVE::Path -- while PVE:

[pve-devel] [PATCH v1 pve-common 1/4] introduce PVE::Path

2024-12-19 Thread Max Carrara
The PVE::Path module concerns itself with file / directory path operations, like getting the parent directory of a path, extracting the file name of a path, splitting a path into its individual components, joining path components together, comparing paths, and so on. This module is added here in o

[pve-devel] [PATCH v1 pve-common 4/4] debian: introduce package libproxmox-fs-path-utils-perl

2024-12-19 Thread Max Carrara
This package contains both PVE::Path and PVE::Filesystem and is added in order to split off those two modules from the rest of libpve-common-perl from the get-go. Both modules are provided in one package because chances are, if one needs stuff from one module, they'll need stuff from the other mod

[pve-devel] [PATCH v1 pve-common 0/4] Introduce and Package PVE::Path & PVE::Filesystem

2024-12-19 Thread Max Carrara
Introduce and Package PVE::Path & PVE::Filesystem = tl;dr is at the bottom. PVE::Path - Concerns itself with file & directory path operations and does not manipulate the filesystem. This module adds a lot of functionalities that more moder

Re: [pve-devel] [PATCH v1 pve-common 2/4] add tests for PVE::Path

2024-12-19 Thread Thomas Lamprecht
Am 19.12.24 um 19:31 schrieb Max Carrara: > This commit adds a plethora of parameterized tests for all functions > of PVE::Path (except aliases). This surmounts to 1050 tests being run > in total. Some of these tests might perhaps be redundant, but the goal > here was to be better safe than sorry a

[pve-devel] [PATCH v1 pve-common 2/4] add tests for PVE::Path

2024-12-19 Thread Max Carrara
This commit adds a plethora of parameterized tests for all functions of PVE::Path (except aliases). This surmounts to 1050 tests being run in total. Some of these tests might perhaps be redundant, but the goal here was to be better safe than sorry and really make sure that nothing slips through. S