[pve-devel] issues with Virtio-SCSI devicde on Proxmox...

2024-08-13 Thread Christian Moser
Hello, I work for VSI (VMS Software Inc) which is porting the OpenVMS operating system to x86. At this point we successfully on various hypervisors, but have some issues on the KVM running on Proxmox. The OpenVMS VM works just fine with SATA disks and it also works with for example virtio-netwo

Re: [pve-devel] Interest in a file manager interface within pve-manager?

2024-08-13 Thread Dominik Csapak
On 7/12/24 00:27, Blythe, Nathan F. - US via pve-devel wrote: Hello, Hi, sorry for the late answer. First thanks for a well written proposal/question. This is what we'd like to see if third-parties are interested in making features/changes/etc. I try to answer the question/proposal to the

[pve-devel] [PATCH v2 qemu-server] remote migration: fix online migration via API clients

2024-08-13 Thread Fiona Ebner
As reported in the community forum [0], when a remote migration request comes in via an API client, the -T flag for Perl is set, so an insecure dependency in a call like unlink() in forward_unix_socket() will fail with: > failed to write forwarding command - Insecure dependency in unlink while >

Re: [pve-devel] Interest in a file manager interface within pve-manager?

2024-08-13 Thread Dominik Csapak
sorry for the noise, it seems i forgot to put you in CC, so now again with that... On 8/13/24 10:35, Dominik Csapak wrote: On 7/12/24 00:27, Blythe, Nathan F. - US via pve-devel wrote: Hello, Hi, sorry for the late answer. First thanks for a well written proposal/question. This is what we

Re: [pve-devel] [PATCH qemu-server] remote migration: fix online migration via API clients

2024-08-13 Thread Fiona Ebner
Am 09.08.24 um 12:17 schrieb Fiona Ebner: > diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm > index e71face4..7a7183e0 100644 > --- a/PVE/QemuMigrate.pm > +++ b/PVE/QemuMigrate.pm > @@ -1095,7 +1095,8 @@ sub phase2 { > die "only UNIX sockets are supported for remote migration\n" >

Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...

2024-08-13 Thread Fiona Ebner
Hi, Am 12.08.24 um 12:40 schrieb Christian Moser: > Hello, > > I work for VSI (VMS Software Inc) which is porting the OpenVMS operating > system to x86. At this point we successfully on various > hypervisors, but have some issues on the KVM running on Proxmox. > > The OpenVMS VM works just fine

[pve-devel] [PATCH docs] pve-network: correct language errors

2024-08-13 Thread Alexander Zeidler
Signed-off-by: Alexander Zeidler --- pve-network.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pve-network.adoc b/pve-network.adoc index acdcf39..434430d 100644 --- a/pve-network.adoc +++ b/pve-network.adoc @@ -447,7 +447,7 @@ slaves in the single logical bonded int

[pve-devel] [PATCH docs] local-zfs: correct language errors

2024-08-13 Thread Alexander Zeidler
Signed-off-by: Alexander Zeidler --- local-zfs.adoc | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/local-zfs.adoc b/local-zfs.adoc index 130f9d6..7ddf50a 100644 --- a/local-zfs.adoc +++ b/local-zfs.adoc @@ -442,7 +442,7 @@ recommend to: # zpool create -f -o as

Re: [pve-devel] [PATCH docs] pve-network: correct language errors

2024-08-13 Thread Lukas Wagner
On 2024-08-13 12:23, Alexander Zeidler wrote: > Signed-off-by: Alexander Zeidler > --- > pve-network.adoc | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/pve-network.adoc b/pve-network.adoc > index acdcf39..434430d 100644 > --- a/pve-network.adoc > +++ b/pve-netw

[pve-devel] [PATCH qemu v2 02/25] PVE backup: fixup error handling for fleecing

2024-08-13 Thread Fiona Ebner
The drained section needs to be terminated before breaking out of the loop in the error scenarios. Otherwise, guest IO on the drive would become stuck. If the job is created successfully, then the job completion callback will clean up the snapshot access block nodes. In case failure happened befor

[pve-devel] [PATCH qemu v2 03/25] PVE backup: factor out setting up snapshot access for fleecing

2024-08-13 Thread Fiona Ebner
Avoids some line bloat in the create_backup_jobs_bh() function and is in preparation for setting up the snapshot access independently of fleecing, in particular that will be useful for providing access to the snapshot via NBD. Signed-off-by: Fiona Ebner --- No changes in v2. pve-backup.c | 95

[pve-devel] [RFC storage v2 11/25] extract backup config: delegate to backup provider if there is one

2024-08-13 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- Changes in v2: * Adapt to method rename. src/PVE/Storage.pm | 10 ++ 1 file changed, 10 insertions(+) diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm index aea57ab..8993ba7 100755 --- a/src/PVE/Storage.pm +++ b/src/PVE/Storage.pm @@ -1726,6 +1726,16

[pve-devel] [PATCH qemu v2 04/25] PVE backup: save device name in device info structure

2024-08-13 Thread Fiona Ebner
The device name needs to be queried while holding the graph read lock and since it doesn't change during the whole operation, just get it once during setup and avoid the need to query it again in different places. Also in preparation to use it more often in error messages and for the upcoming exte

[pve-devel] [PATCH qemu-server v2 14/25] move nbd_stop helper to QMPHelpers module

2024-08-13 Thread Fiona Ebner
Like this nbd_stop() can be called from a module that cannot include QemuServer.pm. Signed-off-by: Fiona Ebner --- No changes in v2. PVE/API2/Qemu.pm | 3 ++- PVE/CLI/qm.pm| 3 ++- PVE/QemuServer.pm| 6 -- PVE/QemuServer/QMPHelpers.pm | 6 ++ 4 f

[pve-devel] [PATCH qemu-server v2 17/25] backup: keep track of block-node size instead of volume size

2024-08-13 Thread Fiona Ebner
For fleecing, the size needs to match exactly with what QEMU sees. In particular, EFI disks might be attached with a 'size=' option, meaning that size can be different from the volume's size. Commit 36377acf ("backup: disk info: also keep track of size") introduced size tracking and it was only use

[pve-devel] [RFC container v2 22/25] backup: implement backup for external providers

2024-08-13 Thread Fiona Ebner
The filesystem structure is made available as a directory in a consistent manner (with details depending on the vzdump backup mode) just like for regular backup via tar. The backup provider needs to back up the guest and firewall configuration and then the filesystem structure, honoring the ID map

[pve-devel] [RFC qemu/storage/qemu-server/container/manager v2 00/25] backup provider API

2024-08-13 Thread Fiona Ebner
Changes in v2: * Add 'block-device' backup mechansim for VMs. The NBD export is mounted by Proxmox VE and only the block device path (as well as a callback to get the next dirty range for bitmaps) is passed to the backup provider. * Add POC example for Borg - note that I tested with borg 1.2.

[pve-devel] [RFC qemu-server v2 21/25] backup: implement restore for external providers

2024-08-13 Thread Fiona Ebner
First, the provider is asked about what restore mechanism to use. Currently, only 'qemu-img' is possible. Then the configuration files are restored, the provider gives information about volumes contained in the backup and finally the volumes are restored via 'qemu-img convert'. The code for the re

[pve-devel] [RFC manager v2 25/25] backup: implement backup for external providers

2024-08-13 Thread Fiona Ebner
Hooks from the backup provider are called during start/end/abort for both job and backup. And it is necessary to adapt some log messages and special case some things like is already done for PBS, e.g. log file handling. Signed-off-by: Fiona Ebner --- Changes in v2: * Adapt to API changes. PVE/

[pve-devel] [POC storage v2 12/25] add backup provider example

2024-08-13 Thread Fiona Ebner
The example uses a simple directory structure to save the backups, grouped by guest ID. VM backups are saved as configuration files and qcow2 images, with backing files when doing incremental backups. Container backups are saved as configuration files and a tar file or squashfs image (added to test

[pve-devel] [RFC storage v2 10/25] plugin: introduce new_backup_provider() method

2024-08-13 Thread Fiona Ebner
The new_backup_provider() method can be used by storage plugins for external backup providers. If the method returns a provider, Proxmox VE will use callbacks to that provider for backups and restore instead of using its usual backup/restore mechanisms. API age and version are both bumped. The ba

[pve-devel] [PATCH qemu v2 01/25] block/reqlist: allow adding overlapping requests

2024-08-13 Thread Fiona Ebner
Allow overlapping request by removing the assert that made it impossible. There are only two callers: 1. block_copy_task_create() It already asserts the very same condition before calling reqlist_init_req(). 2. cbw_snapshot_read_lock() There is no need to have read requests be non-overlapping i

[pve-devel] [PATCH qemu v2 05/25] PVE backup: include device name in error when setting up snapshot access fails

2024-08-13 Thread Fiona Ebner
Signed-off-by: Fiona Ebner --- No changes in v2. pve-backup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pve-backup.c b/pve-backup.c index 33c23e53c2..d931746453 100644 --- a/pve-backup.c +++ b/pve-backup.c @@ -626,7 +626,8 @@ static void create_backup_jobs_bh(void *o

[pve-devel] [RFC qemu v2 06/25] PVE backup: add target ID in backup state

2024-08-13 Thread Fiona Ebner
In preparation for allowing multiple backup providers. Each backup target can then have its own dirty bitmap and there can be additional checks that the current backup state is actually associated to the expected target. Signed-off-by: Fiona Ebner --- No changes in v2. pve-backup.c | 8 +++

[pve-devel] [RFC qemu v2 07/25] PVE backup: get device info: allow caller to specify filter for which devices use fleecing

2024-08-13 Thread Fiona Ebner
For providing snapshot-access to external backup providers, EFI and TPM also need an associated fleecing image. The new caller will thus need a different filter. Signed-off-by: Fiona Ebner --- No changes in v2. pve-backup.c | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff -

[pve-devel] [PATCH qemu-server v2 20/25] restore: die early when there is no size for a device

2024-08-13 Thread Fiona Ebner
Makes it a clean error for buggy (external) backup providers where the size might not be set at all. Signed-off-by: Fiona Ebner --- No changes in v2. PVE/QemuServer.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index e5ff5efb..37f56f69 100644 ---

[pve-devel] [PATCH qemu-server v2 16/25] backup: cleanup: check if VM is running before issuing QMP commands

2024-08-13 Thread Fiona Ebner
When the VM is only started for backup, the VM will be stopped at that point again. While the detach helpers do not warn about errors currently, that might change in the future. This is also in preparation for other cleanup QMP helpers that are more verbose about failure. Signed-off-by: Fiona Ebne

[pve-devel] [PATCH qemu-server v2 15/25] backup: move cleanup of fleecing images to cleanup method

2024-08-13 Thread Fiona Ebner
TPM drives are already detached there and it's better to group these things together. Signed-off-by: Fiona Ebner --- No changes in v2. PVE/VZDump/QemuServer.pm | 25 + 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/PVE/VZDump/QemuServer.pm b/PVE/VZDump/Q

[pve-devel] [PATCH manager v2 24/25] ui: backup: also check for backup subtype to classify archive

2024-08-13 Thread Fiona Ebner
In anticipation of future storage plugins that might not have PBS-specific formats or adhere to the vzdump naming scheme for backups. Signed-off-by: Fiona Ebner --- No changes in v2. www/manager6/Utils.js | 10 ++ www/manager6/grid/BackupView.js| 4 ++-- www/manager6/

[pve-devel] [RFC qemu v2 08/25] PVE backup: implement backup access setup and teardown API for external providers

2024-08-13 Thread Fiona Ebner
For external backup providers, the state of the VM's disk images at the time the backup is started is preserved via a snapshot-access block node. Old data is moved to the fleecing image when new guest writes come in. The snapshot-access block node, as well as the associated bitmap in case of increm

[pve-devel] [RFC qemu-server v2 18/25] backup: allow adding fleecing images also for EFI and TPM

2024-08-13 Thread Fiona Ebner
For the external backup API, it will be necessary to add a fleecing image even for small disks like EFI and TPM, because there is no other place the old data could be copied to when a new guest write comes in. Signed-off-by: Fiona Ebner --- No changes in v2. PVE/VZDump/QemuServer.pm | 14 +

[pve-devel] [RFC qemu v2 09/25] PVE backup: implement bitmap support for external backup access

2024-08-13 Thread Fiona Ebner
There can be one dirty bitmap for each backup target ID (which are tracked in the backup_access_bitmaps hash table). The QMP user can specify the ID of the bitmap it likes to use. This ID is then compared to the current one for the given target. If they match, the bitmap is re-used (should it still

[pve-devel] [POC storage v2 13/25] Borg plugin

2024-08-13 Thread Fiona Ebner
Archive names start with the guest type and ID and then the same timestamp format as PBS. Container archives have the following structure: guest.config firewall.config filesystem/ # containing the whole filesystem structure VM archives have the following structure guest.config firewall.config vol

[pve-devel] [RFC qemu-server v2 19/25] backup: implement backup for external providers

2024-08-13 Thread Fiona Ebner
The state of the VM's disk images at the time the backup is started is preserved via a snapshot-access block node. Old data is moved to the fleecing image when new guest writes come in. The snapshot-access block node, as well as the associated bitmap in case of incremental backup, will be made avai

[pve-devel] [RFC container v2 23/25] backup: implement restore for external providers

2024-08-13 Thread Fiona Ebner
First, the provider is asked about what restore mechanism to use. Currently, 'directory' and 'tar' are possible, for restoring either from a directory containing the full filesystem structure (for which rsync is used) or a potentially compressed tar file containing the same. The new functions are

[pve-devel] applied: [PATCH docs] pve-network: correct language errors

2024-08-13 Thread Fiona Ebner
Am 13.08.24 um 12:23 schrieb Alexander Zeidler: > Signed-off-by: Alexander Zeidler > --- > pve-network.adoc | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/pve-network.adoc b/pve-network.adoc > index acdcf39..434430d 100644 > --- a/pve-network.adoc > +++ b/pve-networ

[pve-devel] applied: [PATCH docs] local-zfs: correct language errors

2024-08-13 Thread Fiona Ebner
Am 13.08.24 um 12:23 schrieb Alexander Zeidler: > Signed-off-by: Alexander Zeidler > --- > local-zfs.adoc | 14 +++--- > 1 file changed, 7 insertions(+), 7 deletions(-) > applied, thanks! Squashed in a change to wrap lines that would've been longer than 80 characters.

Re: [pve-devel] applied: [PATCH docs] pve-network: correct language errors

2024-08-13 Thread Fiona Ebner
Am 13.08.24 um 15:57 schrieb Fiona Ebner: > Am 13.08.24 um 12:23 schrieb Alexander Zeidler: >> Signed-off-by: Alexander Zeidler >> --- >> pve-network.adoc | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/pve-network.adoc b/pve-network.adoc >> index acdcf39..434430d

Re: [pve-devel] [RFC firewall/proxmox{-ve-rs, -firewall, -perl-rs} 00/21] autogenerate ipsets for sdn objects

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > This patch series adds support for autogenerating ipsets for SDN objects. It > autogenerates ipsets for every VNet as follows: > > * ipset containing all IP ranges of the VNet > * ipset containing all gateways of the VNet > * ipset conta

Re: [pve-devel] [PATCH proxmox-ve-rs 01/21] debian: add files for packaging

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > Since we now have a standalone repository for Proxmox VE related > crates, add the required files for packaging the crates contained in > this repository. > > Signed-off-by: Stefan Hanreich > --- > .cargo/config.toml

Re: [pve-devel] [PATCH proxmox-ve-rs 02/21] firewall: add ip range types

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > Currently we are using tuples to represent IP ranges which is > suboptimal. Validation logic and invariant checking needs to happen at > every site using the IP range rather than having a unified struct for > enforcing those invariants.

Re: [pve-devel] [PATCH proxmox-ve-rs 05/21] iprange: add methods for converting an ip range to cidrs

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > This is mainly used in proxmox-perl-rs, so the generated ipsets can be > used in pve-firewall where only CIDRs are supported. > > Signed-off-by: Stefan Hanreich > --- > .../src/firewall/types/address.rs | 818 ++

Re: [pve-devel] [PATCH proxmox-ve-rs 10/21] sdn: add ipam module

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > This module includes structs for representing the JSON schema from the > PVE ipam. Those can be used to parse the current IPAM state. > > We also include a general Ipam struct, and provide a method for > converting the PVE IPAM to the ge

Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...

2024-08-13 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- Hi, Currently they are no way to add custom options for virtio-devices command line from vm config, so it should be patched to add support for openvms os and add special tuning. for example:https://git.proxmox.com/?p=qemu-server.git;a=blob_plain;f=PVE/QemuServer.pm;hb=HEAD

Re: [pve-devel] [PATCH pve-firewall 19/21] add support for loading sdn firewall configuration

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > Signed-off-by: Stefan Hanreich > --- > src/PVE/Firewall.pm | 43 +-- > 1 file changed, 41 insertions(+), 2 deletions(-) > > diff --git a/src/PVE/Firewall.pm b/src/PVE/Firewall.pm > index 09544ba.

Re: [pve-devel] [PATCH proxmox-perl-rs 21/21] add PVE::RS::Firewall::SDN module

2024-08-13 Thread Max Carrara
On Wed Jun 26, 2024 at 2:15 PM CEST, Stefan Hanreich wrote: > Used for obtaining the IPSets that get autogenerated by the nftables > firewall. The returned configuration has the same format as the > pve-firewall uses internally, making it compatible with the existing > pve-firewall code. > > Signed

[pve-devel] [PATCH installer v2 4/5] fix #5250: tui: expose new btrfs `compress` option

2024-08-13 Thread Christoph Heiss
Signed-off-by: Christoph Heiss --- Changes v1 -> v2: * rebased on master proxmox-tui-installer/src/views/bootdisk.rs | 37 + 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/proxmox-tui-installer/src/views/bootdisk.rs b/proxmox-tui-installer/src/views/bootdi

[pve-devel] [PATCH installer v2 1/5] fix #5250: install: config: add new `btrfs_opts` with `compress` config option

2024-08-13 Thread Christoph Heiss
Signed-off-by: Christoph Heiss --- Changes v1 -> v2: * no changes Proxmox/Install/Config.pm | 15 ++ proxmox-auto-installer/src/utils.rs | 1 + proxmox-installer-common/src/options.rs | 31 + proxmox-installer-common/src/setup.rs

[pve-devel] [PATCH installer v2 2/5] fix #5250: install: write btrfs `compress` option to fstab

2024-08-13 Thread Christoph Heiss
`compress` instead of `compress-force` is used, as the latter can have unindented (performance) implications, as the name implies. That would be neither expected by users nor should such a decision made without the user explicitly opting for it. Others do the same, e.g. the installer for RedHat/Fe

[pve-devel] [PATCH installer v2 3/5] fix #5250: proxinstall: expose new btrfs `compress` option

2024-08-13 Thread Christoph Heiss
Signed-off-by: Christoph Heiss --- Changes v1 -> v2: * no changes * moved some mount option setup code to previous patch proxinstall | 15 +++ 1 file changed, 15 insertions(+) diff --git a/proxinstall b/proxinstall index 12f3eaa..c7776f0 100755 --- a/proxinstall +++ b/proxinstal

[pve-devel] [PATCH installer v2 5/5] fix #5250: auto-installer: expose new btrfs `compress` option

2024-08-13 Thread Christoph Heiss
Signed-off-by: Christoph Heiss --- Changes v1 -> v2: * squashed in separate tests patch proxmox-auto-installer/src/answer.rs | 6 - proxmox-auto-installer/src/utils.rs | 6 - .../tests/resources/parse_answer/btrfs.json | 24 +++ .../tests/resource

[pve-devel] [PATCH installer v2 0/5] fix #5250: add btrfs `compress` mount option support

2024-08-13 Thread Christoph Heiss
Fixes #5250 [0]. Pretty much as it says on the tin, this adds the `compress` mount option for Btrfs much in the same way as the ZFS equivalent. W.r.t to the discussion in #5250 - `compress` is used. For a detailed explanation, see patch #2. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5250

Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...

2024-08-13 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- Hi, I didn't see the responde of Fiona but indeed: https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html "virtio devices can be exposed in upto three ways - Legacy - follows virtio 0.9 specification. always uses PCI ID range 0x1000-0x103F - Tran