Re: [pve-devel] [PATCH guest-common/container/qemu-server/manager v2 0/6] fix #4474: stop tasks may overrule shutdown tasks

2024-04-02 Thread Friedrich Weber
ping -- still applies. On 30/01/2024 18:10, Friedrich Weber wrote: > As reported in #4474 [0], a user may attempt to shutdown a VM/CT, > realize that it is unresponsive, and decide to stop it instead. If the > shutdown task has not timed out yet, the stop task will fail. The user > needs to manual

Re: [pve-devel] [RFC container/firewall/manager/proxmox-firewall/qemu-server 00/37] proxmox firewall nftables implementation

2024-04-02 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- >>## Known Issues >>There is currently one major issue that we still need to solve: >>REJECTing >>packets from the guest firewalls is currently not possible for >>incoming traffic >>(it will instead be dropped). That's remember me this old Hetzner bug (Hetzner flooding bad p

Re: [pve-devel] [RFC container/firewall/manager/proxmox-firewall/qemu-server 00/37] proxmox firewall nftables implementation

2024-04-02 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- >> >>## Known Issues >>There is currently one major issue that we still need to solve: >>REJECTing >>packets from the guest firewalls is currently not possible for >>incoming traffic >>(it will instead be dropped). That remember me this Hetzner bug --- End Message --- _

Re: [pve-devel] [RFC container/firewall/manager/proxmox-firewall/qemu-server 00/37] proxmox firewall nftables implementation

2024-04-02 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- Hi Stefan, I'll really take time to test it (I was super busy theses last month with a datacenter migration), as I wait for nftables since a while. Can't help too much with rust, but I really appriciate it, as I had some servers with a lot of vms && rules, take more than 10s

Re: [pve-devel] [RFC container/firewall/manager/proxmox-firewall/qemu-server 00/37] proxmox firewall nftables implementation

2024-04-02 Thread Laurent GUERBY
On Tue, 2024-04-02 at 19:15 +0200, Stefan Hanreich wrote: > > ## Known Issues > There is currently one major issue that we still need to solve: REJECTing > packets from the guest firewalls is currently not possible for incoming > traffic > (it will instead be dropped). > > This is due to the fac

[pve-devel] [PATCH proxmox-firewall 31/37] firewall: add ruleset generation logic

2024-04-02 Thread Stefan Hanreich
We create the rules from the firewall config by utilizing the ToNftRules and ToNftObjects traits to convert the firewall config structs to nftables objects/chains/rules. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-firewall/Cargo.toml | 3 + proxmox-firewa

[pve-devel] [PATCH proxmox-firewall 25/37] nftables: add libnftables bindings

2024-04-02 Thread Stefan Hanreich
Add a thin wrapper around libnftables, which can be used to run commands defined by the rust types. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-nftables/src/context.rs | 243 proxmox-nftables/src/error.rs | 43 ++ proxmox-

[pve-devel] [PATCH proxmox-firewall 22/37] nftables: statement: add conversion traits for config types

2024-04-02 Thread Stefan Hanreich
Some types from the firewall configuration map directly onto nftables statements. For those we implement conversion traits so we can conveniently convert between the configuration types and the respective nftables types. As with the expressions, those are guarded behind a feature so the nftables c

[pve-devel] [PATCH proxmox-firewall 19/37] nftables: expression: add types

2024-04-02 Thread Stefan Hanreich
Adds an enum containing most of the expressions defined in the nftables-json schema [1]. [1] https://manpages.debian.org/bookworm/libnftables1/libnftables-json.5.en.html#EXPRESSIONS Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-nftables/Cargo.toml| 2 +

[pve-devel] [PATCH pve-firewall 36/37] add configuration option for new nftables firewall

2024-04-02 Thread Stefan Hanreich
Introduces new nftables configuration option that en/disables the new nftables firewall. pve-firewall reads this option and only generates iptables rules when nftables is set to `0`. Conversely proxmox-firewall only generates nftables rules when the option is set to `1`. Signed-off-by: Stefan Han

[pve-devel] [PATCH proxmox-firewall 28/37] firewall: add config loader

2024-04-02 Thread Stefan Hanreich
We load the firewall configuration from the default paths, as well as only the guest configurations that are local to the node itself. In the future we could change this to use pmxcfs directly instead. We also load information from nftables directly about dynamically created chains (mostly chains

[pve-devel] [PATCH qemu-server 34/37] firewall: add handling for new nft firewall

2024-04-02 Thread Stefan Hanreich
When the nftables firewall is enabled, we do not need to create firewall bridges. Signed-off-by: Stefan Hanreich --- vm-network-scripts/pve-bridge | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/vm-network-scripts/pve-bridge b/vm-network-scripts/pve-bridge index 8599

[pve-devel] [PATCH proxmox-firewall 16/37] config: firewall: add conntrack helper types

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/resources/ct_helper.json | 52 + proxmox-ve-config/src/firewall/ct_helper.rs | 115 proxmox-ve-config/src/firewall/mod.rs | 1 + 3 files changed, 168 insertions(+) creat

[pve-devel] [PATCH proxmox-firewall 29/37] firewall: add rule generation logic

2024-04-02 Thread Stefan Hanreich
ToNftRules is basically a conversion trait for firewall config structs to convert them into the respective nftables statements. We are passing a list of rules to the method, which then modifies the list of rules such that all relevant rules in the list have statements appended that apply the confi

[pve-devel] [PATCH proxmox-firewall 27/37] firewall: add base ruleset

2024-04-02 Thread Stefan Hanreich
This is the skeleton for the firewall that contains all the base chains required for the firewall. The file applies atomically, which means that it flushes all objects and recreates them - except for the cluster/host/guest chain. This means that it can be run at any point in time, since it only up

[pve-devel] [PATCH pve-manager 37/37] firewall: expose configuration option for new nftables firewall

2024-04-02 Thread Stefan Hanreich
Signed-off-by: Stefan Hanreich --- www/manager6/grid/FirewallOptions.js | 1 + 1 file changed, 1 insertion(+) diff --git a/www/manager6/grid/FirewallOptions.js b/www/manager6/grid/FirewallOptions.js index 0ac9979c4..2d21d45d8 100644 --- a/www/manager6/grid/FirewallOptions.js +++ b/www/manager6/

[pve-devel] [PATCH proxmox-firewall 30/37] firewall: add object generation logic

2024-04-02 Thread Stefan Hanreich
ToNftObjects is basically a conversion trait that converts firewall config structs into nftables objects. It returns a list of commands that create the respective nftables objects. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-firewall/src/main.rs | 1 + proxm

[pve-devel] [PATCH pve-container 35/37] firewall: add handling for new nft firewall

2024-04-02 Thread Stefan Hanreich
When the nftables firewall is enabled, we do not need to create firewall bridges. Signed-off-by: Stefan Hanreich --- src/PVE/LXC.pm | 5 + 1 file changed, 5 insertions(+) diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm index 7883cfb..a5d389a 100644 --- a/src/PVE/LXC.pm +++ b/src/PVE/LXC.pm @@

[pve-devel] [PATCH proxmox-firewall 15/37] config: firewall: add firewall macros

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/resources/macros.json | 896 proxmox-ve-config/src/firewall/fw_macros.rs | 69 ++ proxmox-ve-config/src/firewall/mod.rs | 1 + 3 files changed, 966 insertions(+) create mode

[pve-devel] [PATCH proxmox-firewall 32/37] firewall: add proxmox-firewall binary

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-firewall/src/main.rs | 34 ++ 1 file changed, 34 insertions(+) diff --git a/proxmox-firewall/src/main.rs b/proxmox-firewall/src/main.rs index 53c1289..28aecdf 100644 --- a/proxmox-firewa

[pve-devel] [PATCH proxmox-firewall 20/37] nftables: expression: implement conversion traits for firewall config

2024-04-02 Thread Stefan Hanreich
Some types from the firewall configuration map directly onto nftables expressions. For those we implement conversion traits so we can conveniently convert between the configuration types and the respective nftables types. Those are guarded behind a feature so the nftables crate can be used standal

[pve-devel] [PATCH proxmox-firewall 26/37] firewall: add firewall crate

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- Cargo.toml | 1 + proxmox-firewall/Cargo.toml | 17 + proxmox-firewall/src/main.rs | 5 + 3 files changed, 23 insertions(+) create mode 100644 proxmox-firewall/Cargo.toml create mode 1

[pve-devel] [PATCH proxmox-firewall 13/37] config: firewall: add host specific config + option types

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/host.rs | 309 + proxmox-ve-config/src/firewall/mod.rs | 1 + 2 files changed, 310 insertions(+) create mode 100644 proxmox-ve-config/src/firewall/host.rs diff --git a

[pve-devel] [PATCH proxmox-firewall 14/37] config: firewall: add guest-specific config + option types

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/guest.rs | 194 proxmox-ve-config/src/firewall/mod.rs | 1 + 2 files changed, 195 insertions(+) create mode 100644 proxmox-ve-config/src/firewall/guest.rs diff --git

[pve-devel] [PATCH proxmox-firewall 12/37] config: firewall: add cluster-specific config + option types

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/cluster.rs | 342 ++ proxmox-ve-config/src/firewall/mod.rs | 1 + 2 files changed, 343 insertions(+) create mode 100644 proxmox-ve-config/src/firewall/cluster.rs diff -

[pve-devel] [PATCH proxmox-firewall 02/37] config: firewall: add types for ip addresses

2024-04-02 Thread Stefan Hanreich
Includes types for all kinds of IP values that can occur in the firewall config. Additionally, FromStr implementations are available for parsing from the config files. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/mod.rs | 1 + ...

[pve-devel] [PATCH proxmox-firewall 17/37] nftables: add crate for libnftables bindings

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- Cargo.toml | 1 + proxmox-nftables/Cargo.toml | 16 proxmox-nftables/src/lib.rs | 0 3 files changed, 17 insertions(+) create mode 100644 proxmox-nftables/Cargo.toml create mode 100644 prox

[pve-devel] [PATCH proxmox-firewall 05/37] config: firewall: add types for aliases

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/parse.rs | 44 + proxmox-ve-config/src/firewall/types/alias.rs | 160 ++ proxmox-ve-config/src/firewall/types/mod.rs | 2 + 3 files changed, 206 insertions(+) creat

[pve-devel] [PATCH proxmox-firewall 09/37] config: firewall: add types for rules

2024-04-02 Thread Stefan Hanreich
Additionally we implement FromStr for all rule types and parts, which can be used for parsing firewall config rules. Initial rule parsing works by parsing the different options into a HashMap and only then de-serializing a struct from the parsed options. This intermediate step makes rule parsing a

[pve-devel] [PATCH proxmox-firewall 23/37] nftables: commands: add types

2024-04-02 Thread Stefan Hanreich
Add rust types for most of the nftables commands as defined by libnftables-json [1]. Different commands require different keys to be set for the same type of object. E.g. deleting an object usually only requires a name + name of the container (table/chain/rule). Creating an object usually requires

[pve-devel] [PATCH proxmox-firewall 33/37] firewall: add files for debian packaging

2024-04-02 Thread Stefan Hanreich
Signed-off-by: Stefan Hanreich --- Makefile| 93 + debian/changelog| 5 ++ debian/control | 31 +++ debian/copyright| 16 ++ debian/proxmox-firewall.service | 16 ++ debian/pr

[pve-devel] [PATCH proxmox-firewall 21/37] nftables: statement: add types

2024-04-02 Thread Stefan Hanreich
Adds an enum containing most of the statements defined in the nftables-json schema [1]. [1] https://manpages.debian.org/bookworm/libnftables1/libnftables-json.5.en.html#STATEMENTS Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-nftables/Cargo.toml | 1 + p

[pve-devel] [PATCH proxmox-firewall 04/37] config: firewall: add types for log level and rate limit

2024-04-02 Thread Stefan Hanreich
Adds types for log and (log-)rate-limiting firewall config options as well as FromStr implementations for parsing them from the config. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/Cargo.toml| 1 + proxmox-ve-config/src/firewall/mod.rs

[pve-devel] [PATCH proxmox-firewall 03/37] config: firewall: add types for ports

2024-04-02 Thread Stefan Hanreich
Adds types for all kinds of port-related values in the firewall config as well as FromStr implementations for parsing them from the config. Also adds a helper for parsing the named ports from `/etc/services`. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-confi

[pve-devel] [PATCH proxmox-firewall 07/37] config: guest: add helpers for parsing guest network config

2024-04-02 Thread Stefan Hanreich
Currently this is parsing the config files via the filesystem. In the future we could also get this information from pmxcfs directly via IPC which should be more performant, particularly for a large number of VMs. Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-c

[pve-devel] [PATCH proxmox-firewall 24/37] nftables: types: add conversion traits

2024-04-02 Thread Stefan Hanreich
Some parts of the firewall config map directly to nftables objects, so we introduce conversion traits for convenient conversion into the respective nftables objects / types. They are guarded behind a feature, so the nftables crate can be used standalone without depending on the proxmox-ve-config c

[pve-devel] [PATCH proxmox-firewall 18/37] nftables: add helpers

2024-04-02 Thread Stefan Hanreich
Several objects, statements and expressions in nftables-json require null values, for instance: { "flush": { "ruleset": null }} For this purpose we define our own Null type, which we can then easily use for defining types that accept Null as value. Several keys accept as value either a singu

[pve-devel] [PATCH proxmox-firewall 11/37] config: firewall: add generic parser for firewall configs

2024-04-02 Thread Stefan Hanreich
Since the basic format of cluster, host and guest firewall configurations is the same, we create a generic parser that can handle the common config format. The main difference is in the available options, which can be passed via a generic parameter. Co-authored-by: Wolfgang Bumiller Signed-off-by

[pve-devel] [PATCH proxmox-firewall 08/37] config: firewall: add types for ipsets

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/types/ipset.rs | 345 ++ proxmox-ve-config/src/firewall/types/mod.rs | 2 + 2 files changed, 347 insertions(+) create mode 100644 proxmox-ve-config/src/firewall/types/ipset.rs

[pve-devel] [PATCH proxmox-firewall 06/37] config: host: add helpers for host network configuration

2024-04-02 Thread Stefan Hanreich
Currently the helpers for obtaining the host network configuration panic on error, which could be avoided by the use of OnceLock::get_or_init, but this method is currently only available in nightly versions. Generally, if there is a problem with obtaining a hostname for the current node then somet

[pve-devel] [PATCH proxmox-firewall 10/37] config: firewall: add types for security groups

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- proxmox-ve-config/src/firewall/types/group.rs | 36 +++ proxmox-ve-config/src/firewall/types/mod.rs | 2 ++ 2 files changed, 38 insertions(+) create mode 100644 proxmox-ve-config/src/firewall/types/group.rs

[pve-devel] [RFC container/firewall/manager/proxmox-firewall/qemu-server 00/37] proxmox firewall nftables implementation

2024-04-02 Thread Stefan Hanreich
## Introduction This RFC provides a drop-in replacement for the current pve-firewall package that is based on Rust and nftables. It consists of three crates: * proxmox-ve-config for parsing firewall and guest configuration files, as well as some helpers to access host configuration (particular

[pve-devel] [PATCH proxmox-firewall 01/37] config: add proxmox-ve-config crate

2024-04-02 Thread Stefan Hanreich
Co-authored-by: Wolfgang Bumiller Signed-off-by: Stefan Hanreich --- .cargo/config| 5 + .gitignore | 6 ++ Cargo.toml | 4 proxmox-ve-config/Cargo.toml | 19 +++ proxmox-ve-config/src/lib.rs | 0 5 files changed

Re: [pve-devel] [PATCH pve-network 0/8] SDN Vnet blackbox testing

2024-04-02 Thread Stefan Lendl
sent v2 incorporating the changes and adding additional tests. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 pve-network 7/9] api: extract function that creates the sdn directory.

2024-04-02 Thread Stefan Lendl
create_etc_interfaces_sdn_dir creates the /etc/pve/sdn directory. This allows mocking in tests to prevent system fs access in tests Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/API2/Network/SDN/Zones.pm | 6 +- 1 file changed, 5 insertions(+), 1 d

[pve-devel] [PATCH v2 pve-network 4/9] dnsmasq: extract function that updates dnsmasq lease via dbus

2024-04-02 Thread Stefan Lendl
Extract the dbus based interactions with dnsmasq so that it can be mocked in tests. Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Dhcp/Dnsmasq.pm | 22 -- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/s

[pve-devel] [PATCH v2 pve-network 1/9] sdn: extract function that reads datacenter config

2024-04-02 Thread Stefan Lendl
The datacenter_config() functions in SDN::Zones::Plugin is a simple wrapper that reads datacenter.cfg via cfs. This allows mocking datacenter.cfg in tests. Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Zones/EvpnPlugin.pm | 3 ++- src/PVE

[pve-devel] [PATCH v2 pve-network 2/9] dnsmasq: extract function to systemctl command.

2024-04-02 Thread Stefan Lendl
systemctl_service() is a wrapper around PVE::Tools::run_command to allow mocking the systemctl interactions in tests. Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Dhcp/Dnsmasq.pm | 18 -- 1 file changed, 12 insertions(+), 6

[pve-devel] [PATCH v2 pve-network 3/9] dnsmasq: extract function that generates the ethers file path

2024-04-02 Thread Stefan Lendl
Extracted to a function so it can be mocked in tests. Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Dhcp/Dnsmasq.pm | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/PVE/Network/SDN/Dhcp/Dnsmasq.pm b/src/PVE/Net

[pve-devel] [PATCH v2 pve-network 8/9] tests: test VNets functionality as a blackbox

2024-04-02 Thread Stefan Lendl
Add several tests for Vnets in test_vnets_blackbox. State setup as well as testing results is done only via the API to test on the API boundaries not not against the internal state. Internal state is mocked to avoid requiring access to system files or pmxcfs. Mocking is done by reading and writing

[pve-devel] [PATCH v2 pve-network 9/9] tests: remove old Vnets tests

2024-04-02 Thread Stefan Lendl
The did not work and were primarily testing against internal state. Signed-off-by: Stefan Lendl --- src/test/run_test_vnets.pl | 343 - 1 file changed, 343 deletions(-) delete mode 100755 src/test/run_test_vnets.pl diff --git a/src/test/run_test_vnets.pl b/s

[pve-devel] [PATCH v2 pve-network 6/9] evpn: extract function that reads frr config.

2024-04-02 Thread Stefan Lendl
read_local_frr_config reads /etc/frr/frr.conf.local. This allows mocking local fs access in tests. Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) dif

[pve-devel] [PATCH v2 pve-network 0/9] SDN: Testing VNets as a blackbox.

2024-04-02 Thread Stefan Lendl
This add several tests for SDN VNets. State setup as well as testing results is done only via the API to test on the API boundaries and not against the internal state. Internal state and config files are mocked to avoid requiring access to system files or pmxcfs. The first 7 commits extract variou

[pve-devel] [PATCH v2 pve-network 5/9] controllers: extract function that reads network intreaces config

2024-04-02 Thread Stefan Lendl
read_etc_network_interfaces reads /etc/network/interfaces. This allows mocking access to local fs. Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Controllers.pm | 16 +++- 1 file changed, 11 insertions(+), 5 deletions(-) diff --

[pve-devel] [PATCH v5 pve-storage 08/11] cephconfig: align our parser with Ceph's parser

2024-04-02 Thread Max Carrara
This commit rewrites the entire parser for ceph.conf, aligning its behaviour as closely as possible with Ceph's parser grammar [0]. The most notable improvements are as follows: 1. The characters '#' and ';' now both mark comments, instead of just the '#' character. 2. Any character, in

[pve-devel] [PATCH v5 pve-manager 11/11] bin/make: gather helper scripts in separate variable

2024-04-02 Thread Max Carrara
Signed-off-by: Max Carrara --- Changes v2 --> v3: * new Changes v3 --> v4: * none Changes v4 --> v5: * none bin/Makefile | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/bin/Makefile b/bin/Makefile index b221e4b1..180a91b5 100644 --- a/bin/Makefile +++ b/bin/Mak

[pve-devel] [PATCH v5 pve-manager 10/11] fix #4759: ceph: configure ceph-crash.service and its key

2024-04-02 Thread Max Carrara
Due to Ceph dropping privileges when running the 'ceph-crash' daemon [0], it is necessary to allow the daemon to authenticate with its cluster in a safe manner. In order to avoid exposing sensitive keyrings or somehow escalating its privileges again, 'ceph-crash' is therefore provided with its own

[pve-devel] [PATCH v5 pve-storage 07/11] cephconfig: escape un-escaped comment literals on write

2024-04-02 Thread Max Carrara
in order to prevent configuration errors or the configuration being misinterpreted. Signed-off-by: Max Carrara --- Changes v3 --> v4: * new Changes v4 --> v5: * escape *all* comment literals (that are not escaped) instead of only escaping them within values src/PVE/CephConfig.pm | 3 ++

[pve-devel] [PATCH v5 pve-manager 09/11] ceph: introduce '/etc/pve/ceph'

2024-04-02 Thread Max Carrara
This commit adds the '/etc/pve/ceph' directory to our overall expected Ceph configuration. This directory is meant to store cluster-wide, non-private configuration files used by Ceph applications and services that are executed with lower privileges, such as 'ceph-crash.service'. The existence of

[pve-devel] [PATCH v5 pve-storage 06/11] cephconfig: align written key-value pairs by tab

2024-04-02 Thread Max Carrara
instead of tab + space. Signed-off-by: Max Carrara --- Changes v2 --> v3: * new Changes v3 --> v4: * rebased due to previous changes Changes v4 --> v5: (originally patch 12) * rebase onto changes of patch 01 * align by tab only instead of removing all leading whitespace - as it turn

[pve-devel] [PATCH v5 pve-storage 05/11] cephconfig: change order of written sections

2024-04-02 Thread Max Carrara
in order to group related sections together. Additionally, sections that are associated with the client, for example '[client.foo]', are written directly after the '[client]' section. Signed-off-by: Max Carrara --- Changes v2 --> v3: * new; originally patch 07 of series v2, now adapted to this

[pve-devel] [PATCH v5 pve-storage 04/11] cephconfig: allow writing arbitrary sections

2024-04-02 Thread Max Carrara
This adds support for writing arbitrary sections to 'ceph.conf' while ensuring that already written sections are not duplicated. Signed-off-by: Max Carrara --- Changes v1 --> v2: * Instead of just adding 'client.crash' as a separate section, also allow writing arbitrary sections Changes v2

[pve-devel] [PATCH v5 pve-storage 03/11] test: cephconfig: add regression test for Ceph config parser & writer

2024-04-02 Thread Max Carrara
Building on the previously declared cases, this test constructs a config hash akin to the one the parser returns by invoking the `ceph-conf` command. The cases which `ceph-conf` cannot handle are marked with a special key. If the key is provided, invocatinos to `ceph-conf` are expected to fail. Fu

[pve-devel] [PATCH v5 pve-storage, pve-manager 00/11] Fix #4759: Configure Permissions for ceph-crash.service

2024-04-02 Thread Max Carrara
Fix #4759: Configure Permissions for ceph-crash.service - Version 5 === Notable changes since v4 * The patches regarding Ceph Reef and Ceph Quincy have been applied in v4 and are thus dropped from this

[pve-devel] [PATCH v5 pve-storage 02/11] test: add tests for 'ceph.conf' parser and writer

2024-04-02 Thread Max Carrara
These tests attempt to cover most of Ceph's config parser's grammar, including all of its syntax quirks [0]. Each case is tested against two testing subroutines: 1. The parser's output is compared with the expected output. 2. The writer's output is parsed again ant then compared with the

[pve-devel] [PATCH v5 pve-storage 01/11] cephconfig: change code style inside config writer

2024-04-02 Thread Max Carrara
This commit changes the code style of subroutine `write_ceph_config` to match our style guide [0] more. Furthermore, the repeated calls to the inner subroutine are replaced with a loop, while the regular expressions used by the inner `sub` are now quoted with `qr` to prevent any accidental mis-quo

Re: [pve-devel] [PATCH v3 00/30] add automated/unattended installation

2024-04-02 Thread Aaron Lauterer
thanks for looking into this On 2024-04-02 16:43, Christoph Heiss wrote: Rebuild an (PVE) ISO to contain the new auto-installer with an appropriate GRUB entry setting `proxauto` on the kernel commandline. What I've tried: - Using a few different values for `global` options - Install on ext4,

Re: [pve-devel] [PATCH v3 00/30] add automated/unattended installation

2024-04-02 Thread Christoph Heiss
Rebuild an (PVE) ISO to contain the new auto-installer with an appropriate GRUB entry setting `proxauto` on the kernel commandline. What I've tried: - Using a few different values for `global` options - Install on ext4, xfs, Btrfs RAID1 and ZFS RAID1 (with different values in multiple runs) - U

Re: [pve-devel] [PATCH pve-manager v2] sdn: evpn: allow empty primary exit node in zone form

2024-04-02 Thread DERUMIER, Alexandre via pve-devel
--- Begin Message --- Hi, could it be possible to merge this patch ? I have see another report about it on the forum: https://forum.proxmox.com/threads/bugfix-for-evpn-sdn-multiple-exit-nodes.137784/post-649071 Message initial De: Stefan Hanreich Répondre à: Proxmox VE devel

Re: [pve-devel] [PATCH v3 15/30] auto-installer: add fetch answer binary

2024-04-02 Thread Christoph Heiss
Two typos ;) And some small nits On Thu, Mar 28, 2024 at 02:50:13PM +0100, Aaron Lauterer wrote: [..] > diff --git a/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs > b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs > new file mode 100644 > index 000..9e89a3c > --- /dev/null >

Re: [pve-devel] [PATCH v2 pve-network 0/5] SDN tests in sbuild

2024-04-02 Thread Stefan Lendl
Thanks for the review. I rebased the commits for v3 I re-enabled DNS. I do not recall why it failed on my last rebase attempt. ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v3 pve-network 3/5] tests: mocking more functions to avoid system access

2024-04-02 Thread Stefan Lendl
previously extracted functions are now mocked in the zone tests Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/test/run_test_zones.pl | 31 +++ 1 file changed, 31 insertions(+) diff --git a/src/test/run_test_zones.pl b/src/test/

[pve-devel] [PATCH v3 pve-network 5/5] gitignore: build artifacts from sbuild

2024-04-02 Thread Stefan Lendl
source package *.dsc build log *.build tarball *.tar.xz Signed-off-by: Stefan Lendl --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 60332cc..4694865 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,8 @@ *.deb +/*.build /*.buildinfo /*.cha

[pve-devel] [PATCH v3 pve-network 4/5] tests: run tests in sbuild

2024-04-02 Thread Stefan Lendl
Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/Makefile b/src/Makefile index c9dee4c..c4056b4 100644 --- a/src/Makefile +++ b/src/Makefile @@ -10,7 +10,7 @@ clean: .PHONY: te

[pve-devel] [PATCH v3 pve-network 1/5] controllers: extract read_etc_network_interfaces

2024-04-02 Thread Stefan Lendl
to allow mocking local fs access Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Controllers.pm | 16 +++- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/src/PVE/Network/SDN/Controllers.pm b/src/PVE/Network/SDN/Co

[pve-devel] [PATCH v3 pve-network 2/5] evpn: extract read_local_frr_config

2024-04-02 Thread Stefan Lendl
to allow mocking local fs access Signed-off-by: Stefan Lendl Reviewed-by: Max Carrara Tested-by: Max Carrara --- src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/src/PVE/Network/SDN/Controllers/EvpnPlugin.pm b/src/PVE

[pve-devel] [PATCH v3 pve-network 0/5] SDN: Add mocking to run tests in sbuild

2024-04-02 Thread Stefan Lendl
Extract and mock functions that otherwise access system files which is not possible in a clean sbuild environment. Namely /etc/network/interfaces as well as /etc/frr/frr.config.local Added .gitignore for sbuild artifacts Changes v2 -> v3: * Changed commit messages according to commit guide lines *

Re: [pve-devel] [PATCH qemu-server 3/3] api: include not mapped resources for running vms in migrate preconditions

2024-04-02 Thread Dominik Csapak
On 3/22/24 17:19, Fiona Ebner wrote: Am 20.03.24 um 13:51 schrieb Dominik Csapak: so that we can show a proper warning in the migrate dialog and check it in the bulk migrate precondition check the unavailable_storages and allowed_nodes should be the same as before Signed-off-by: Dominik Csapak

Re: [pve-devel] [PATCH qemu-server 4/4] api: enable live migration for marked mapped pci devices

2024-04-02 Thread Dominik Csapak
On 3/22/24 14:37, Fiona Ebner wrote: Am 18.03.24 um 12:18 schrieb Dominik Csapak: They have to be marked as 'live-migration-capable' in the mapping config, and the driver and qemu must support it. For the gui checks, we now return a list of 'mapped-with-live-migration' entries in the migration

Re: [pve-devel] [PATCH qemu-server 3/4] check_local_resources: add more info per mapped device

2024-04-02 Thread Dominik Csapak
On 3/22/24 14:37, Fiona Ebner wrote: Am 18.03.24 um 12:18 schrieb Dominik Csapak: such as the mapping name and if it's marked for live-migration (pci only) Signed-off-by: Dominik Csapak --- PVE/API2/Qemu.pm | 2 +- PVE/QemuMigrate.pm | 5 +++-- PVE/QemuServer.pm | 10 ++ 3 fi

Re: [pve-devel] [PATCH guest-common 2/2] mapping: pci: optionally return the config in 'find_on_current_node'

2024-04-02 Thread Dominik Csapak
On 3/22/24 14:38, Fiona Ebner wrote: Am 18.03.24 um 12:18 schrieb Dominik Csapak: this is useful to get to the config without having to parse it again You could also adapt the call sites that need it to use PVE::Mapping::PCI::config() and PVE::Mapping::PCI::get_node_mapping() instead of PVE::

Re: [pve-devel] [PATCH guest-common 1/2] mapping: pci: add 'live-migration-capable' flag to mappings

2024-04-02 Thread Dominik Csapak
On 3/22/24 14:37, Fiona Ebner wrote: Am 18.03.24 um 12:18 schrieb Dominik Csapak: so that we can decide in qemu-server to allow live-migration. the driver and qemu must be capable of that, and it's the admins responsibility to know and configure that Nit: "The" and "QEMU" should be capitalize

Re: [pve-devel] GET /access/users/{userid} has parameter 'tokens' with 'additionalProperties' containing object definition

2024-04-02 Thread Wolfgang Bumiller
On Tue, Apr 02, 2024 at 09:27:57AM +0200, Fabian Grünbichler wrote: > > > Jona Draaijer via pve-devel hat am 01.04.2024 > > 22:00 CEST geschrieben: > > Hi, > > > > As per the title, that endpoint has an additionalProperties value that is > > not a bool, but rather an object definition. (It's de

Re: [pve-devel] GET /access/users/{userid} has parameter 'tokens' with 'additionalProperties' containing object definition

2024-04-02 Thread Fabian Grünbichler
> Jona Draaijer via pve-devel hat am 01.04.2024 > 22:00 CEST geschrieben: > Hi, > > As per the title, that endpoint has an additionalProperties value that is > not a bool, but rather an object definition. (It's defined in > pve-access-control/src/PVE/API2/User.pm). > > As far as I can tell, a