Re: [pve-devel] [WIP v2 cluster/network/manager/qemu-server/container 00/10] Add support for DHCP servers to SDN

2023-10-17 Thread DERUMIER, Alexandre
Message initial De: Stefan Hanreich À: Proxmox VE development discussion , "DERUMIER, Alexandre" Objet: Re: [pve-devel] [WIP v2 cluster/network/manager/qemu- server/container 00/10] Add support for DHCP servers to SDN Date: 17/10/2023 18:05:55 > Maybe try to see if we can use p

Re: [pve-devel] [RFC] towards automated integration testing

2023-10-17 Thread Thomas Lamprecht
Am 17/10/2023 um 14:33 schrieb Lukas Wagner: > On 10/17/23 08:35, Thomas Lamprecht wrote: >>  From top of my head I'd rather do some attribute based dependency >> annotation, so that one can depend on single tests, or whole fixture >> on others single tests or whole fixture. >> > > The more though

[pve-devel] [PATCH v3 storage 1/1] fix #254: iscsi: add support for multipath iscsi targets

2023-10-17 Thread Yuri Konotopov via pve-devel
--- Begin Message --- Changes since v2: * custom configuration file is removed in favor of open-iscsi query * restored `portal` property format since we should not rely on initial configuration but should use discovered configuration instead * changed check_connection() to query all available

[pve-devel] [PATCH v3 storage 1/1] fix #254: iscsi: add support for multipath iscsi targets

2023-10-17 Thread Yuri Konotopov via pve-devel
--- Begin Message --- With this patch Proxmox now tries to login to all discovered portals in case some of them are not logged yet. In case of multipath configuration when initially configured portal is missing for some reason Proxmox don't lose iscsi storage now and can succesfully restore iscsi c

Re: [pve-devel] [WIP v2 cluster/network/manager/qemu-server/container 00/10] Add support for DHCP servers to SDN

2023-10-17 Thread Stefan Hanreich
> Maybe try to see if we can use pve ipam as cache in front of external > ipam. Yes, it would also be cool if you could look at implementing the two newly added methods from the PVEPlugin for Netbox / Phpipam, since you have more experience with those. I also looked into merging those two methods

Re: [pve-devel] [WIP v2 cluster/network/manager/qemu-server/container 00/10] Add support for DHCP servers to SDN

2023-10-17 Thread Stefan Hanreich
Some additional things we've discussed off-list: Currently for VMs Migration & Hibernation are not working - everything else in the lifecycle of VMs/CTs should be covered. For Migration: It currently creates an additional mapping in the IPAM and doesn't delete the existing mapping from the DHCP

Re: [pve-devel] [WIP v2 cluster/network/manager/qemu-server/container 00/10] Add support for DHCP servers to SDN

2023-10-17 Thread DERUMIER, Alexandre
Hi Stefan, Thanks for sharing ! I'll try to deeply test it this week or next week. Maybe try to see if we can use pve ipam as cache in front of external ipam. Message initial De: Stefan Hanreich Répondre à: Proxmox VE development discussion À: pve-devel@lists.proxmox.com O

[pve-devel] [PATCH installer v2] fix #4869: Show state in management interface ComboBox

2023-10-17 Thread Maximiliano Sandoval R
From: Maximiliano Sandoval This is a continuation of https://lists.proxmox.com/pipermail/pve-devel/2023-August/058639.html. Signed-off-by: Maximiliano Sandoval R --- proxinstall | 23 +-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/proxinstall b/proxinsta

[pve-devel] [WIP v2 pve-network 07/10] dhcp: regenerate config for DHCP servers on reload

2023-10-17 Thread Stefan Hanreich
Regenerate the configuration files for the different DHCP server plugins when applying SDN settings by calling the respective hooks of the plugin responsible for configuring a DHCP instance. Signed-off-by: Stefan Hanreich --- src/PVE/Network/SDN.pm | 11 +- src/PVE/Network/SDN/Dhcp.pm |

[pve-devel] [WIP v2 pve-container 10/10] sdn: dhcp: setup DHCP mappings in LXC hooks

2023-10-17 Thread Stefan Hanreich
Setup DHCP mappings if a container has interfaces on a SDN network managed via DHCP. Additionally remove the mapping in the stop_cleanup function so the mapping gets removed when forcefully stopping the container. Signed-off-by: Stefan Hanreich --- src/PVE/LXC.pm| 10 ++ src

[pve-devel] [WIP v2 cluster/network/manager/qemu-server/container 00/10] Add support for DHCP servers to SDN

2023-10-17 Thread Stefan Hanreich
This is a WIP patch series, since I will be gone for 3 weeks and wanted to share my current progress with the DHCP support for SDN. This patch series adds support for automatically deploying dnsmasq as a DHCP server to a simple SDN Zone. While certainly not 100% polished on some ends (looking at

[pve-devel] [WIP v2 pve-network 03/10] dhcp: add abstract class for DHCP plugins

2023-10-17 Thread Stefan Hanreich
This abstract class provides several hooks that should be called during the config generation process, they expose the functionality for the different configuration tasks required from the DHCP plugins. add_ip_mapping Adds a mapping from MAC address to an IP for a given DHCP server. The DHCP serve

[pve-devel] [WIP v2 pve-network 04/10] dhcp: subnet: add DHCP options to subnet configuration

2023-10-17 Thread Stefan Hanreich
Parse the dhcp-ranges when getting the configuration via the Subnet class. Signed-off-by: Stefan Hanreich --- src/PVE/Network/SDN/SubnetPlugin.pm | 32 + src/PVE/Network/SDN/Subnets.pm | 18 2 files changed, 50 insertions(+) diff --git a/src/PVE

[pve-devel] [WIP v2 pve-cluster 01/10] cluster files: add dhcp.cfg

2023-10-17 Thread Stefan Hanreich
Signed-off-by: Stefan Hanreich --- src/PVE/Cluster.pm | 1 + src/pmxcfs/status.c | 1 + 2 files changed, 2 insertions(+) diff --git a/src/PVE/Cluster.pm b/src/PVE/Cluster.pm index cfa2583..aac4574 100644 --- a/src/PVE/Cluster.pm +++ b/src/PVE/Cluster.pm @@ -78,6 +78,7 @@ my $observed = { '

[pve-devel] [WIP v2 pve-network 05/10] dhcp: add DHCP plugin for dnsmasq

2023-10-17 Thread Stefan Hanreich
The plugin generates several dnsmasq configuration files from the SDN configuration. /etc/default/dnsmasq. This file specifies the configuration directory for the dnsmasq instance (/etc/dnsmasq.d/). It also sets the configuration file to /dev/null so the default configuration from the package has

[pve-devel] [WIP v2 qemu-server 09/10] sdn: dhcp: add DHCP setup to vm-network-scripts

2023-10-17 Thread Stefan Hanreich
When setting up the bridge for the VMs, also set up the DHCP mappings in the respective DHCP plugins if the VM has interfaces on SDN networks that utilize DHCP. Also remove the mapping in the VM cleanup function, so the mappings also get removed when stopping the VM forcefully. Signed-off-by: Ste

[pve-devel] [WIP v2 pve-network 06/10] ipam: Add helper methods for DHCP to PVE IPAM

2023-10-17 Thread Stefan Hanreich
Those methods are used by the DHCP plugins to attain the next free IP address for a given DHCP range, as well as delete all entries with a certain MAC address. Signed-off-by: Stefan Hanreich --- src/PVE/Network/SDN/Ipams/PVEPlugin.pm | 64 ++ 1 file changed, 64 insertions

[pve-devel] [WIP v2 pve-manager 08/10] sdn: regenerate DHCP config on reload

2023-10-17 Thread Stefan Hanreich
Signed-off-by: Stefan Hanreich --- PVE/API2/Network.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/PVE/API2/Network.pm b/PVE/API2/Network.pm index 00d964a79..f39f04f52 100644 --- a/PVE/API2/Network.pm +++ b/PVE/API2/Network.pm @@ -660,6 +660,7 @@ __PACKAGE__->register_method({

[pve-devel] [WIP v2 pve-network 02/10] subnets: vnets: preparations for DHCP plugins

2023-10-17 Thread Stefan Hanreich
Add the option to retrieve the running configuration instead of only the pending configuration via the config methods. Refactor methods using the running config to utilize the new parameter. Additionally include the id for subnets, since it is needed later by some of the DHCP plugins. Signed-off-

[pve-devel] [RFC pve-container] backup: do not delete not backed-up mps on restore

2023-10-17 Thread Christian Ebner
The current behaviour of the restore is to recreate all backed up mountpoints and remove all previous ones, causing potential data loss on restore when the mountpoint was not included in the backup and the user not aware of this behaviour. By checking the mountpoint configuration from the backup,

Re: [pve-devel] [RFC] towards automated integration testing

2023-10-17 Thread Lukas Wagner
On 10/17/23 08:35, Thomas Lamprecht wrote: Is the order of test-cases guaranteed by toml parsing, or how are intra- fixture dependencies ensured? Good point. With rollbacks in between test cases it probably does not matter much, but on 'real hardware' with no rollback this could definitely

Re: [pve-devel] [PATCH-SERIES v2 qemu] update to QEMU 8.1.1

2023-10-17 Thread Fiona Ebner
Superseded by [PATCH-SERIES v3 qemu] update to QEMU 8.1.2: https://lists.proxmox.com/pipermail/pve-devel/2023-October/059485.html ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v3 qemu 5/7] add patch to disable graph locking

2023-10-17 Thread Fiona Ebner
There are still some issues with graph locking, e.g. deadlocks during backup canceling [0] and initial attempts to fix it didn't work [1]. Because the AioContext locks still exist, it should still be safe to disable graph locking. [0]: https://lists.nongnu.org/archive/html/qemu-devel/2023-09/msg00

[pve-devel] [PATCH v3 qemu 6/7] add patch to avoid huge snapshot performance regression

2023-10-17 Thread Fiona Ebner
Taking a snapshot became prohibitively slow because of the migration_transferred_bytes() call in migration_rate_exceeded() [0]. This also applied to the async snapshot taking in Proxmox VE, so work around the issue until it is fixed upstream. [0]: https://gitlab.com/qemu-project/qemu/-/issues/182

[pve-devel] [PATCH v3 qemu 7/7] d/control: add versioned Breaks for qemu-server <= 8.0.6

2023-10-17 Thread Fiona Ebner
Upstream QEMU commit 4271f40383 ("virtio-net: correctly report maximum tx_queue_size value") made setting an invalid tx_queue_size for a non-vDPA/vhost-user net device a hard error. Now, qemu-server before commit 089aed81 ("cfg2cmd: netdev: fix value for tx_queue_size") did just that, so the newer

[pve-devel] [PATCH v3 qemu 1/7] d/rules: use disable-download option instead of git-submodules=ignore

2023-10-17 Thread Fiona Ebner
See the following QEMU commits for reference: 0c5f3dcbb2 ("configure: add --enable-pypi and --disable-pypi") ac4ccac740 ("configure: rename --enable-pypi to --enable-download, control subprojects too") 6f3ae23b29 ("configure: remove --with-git-submodules=") removed The last one removed the option

[pve-devel] [PATCH v3 qemu 2/7] buildsys: fixup submodule target

2023-10-17 Thread Fiona Ebner
It's not enough to initialize the submodules anymore, as some got replaced by wrap files, see QEMU commit 2019cabfee ("meson: subprojects: replace submodules with wrap files"). Download the subprojects during initialization of the QEMU submodule, so building (without the automagical --enable-downl

[pve-devel] [PATCH-SERIES v3 qemu] update to QEMU 8.1.2

2023-10-17 Thread Fiona Ebner
Patch changes: For backup, opening the backup dump block driver needed to be adapted, because of coroutine context changes. Block graph locking was disabled, because of deadlocks. Snapshot code has a huge performance regression which required a workaround. Meta-changes: Use --disable-download

Re: [pve-devel] [RFC] towards automated integration testing

2023-10-17 Thread Thomas Lamprecht
Am 16/10/2023 um 17:18 schrieb Lukas Wagner: > On 10/16/23 13:20, Stefan Hanreich wrote: >> I can imagine having to setup VMs inside the Test Setup as well for >> doing various tests. Doing this manually every time could be quite >> cumbersome / hard to automate. Do you have a mechanism in mind to

Re: [pve-devel] [PATCH v2 storage 1/1] fix #254: iscsi: allow to configure multiple portals

2023-10-17 Thread Yuri Konotopov via pve-devel
--- Begin Message --- 16.10.2023 17:58, Dominik Csapak пишет: i would favor an approach that uses an 'array' instead of the '-list' types. While working more on this patch I came to conclusion we don't need to modify portal property format at all (nor -list, nor -array). What we need for

Re: [pve-devel] [PATCH v2 many 00/11] notifications: feed system mails into proxmox_notify

2023-10-17 Thread Lukas Wagner
Ping - would be great to get some reviews on this to get this merged for the next release. On 10/2/23 10:06, Lukas Wagner wrote: The aim of this patch series is to adapt `proxmox-mail-forward` so that it forwards emails that were sent to the local root user through the `proxmox_notify` crate.

Re: [pve-devel] [PATCH v3 many 0/7] notifications: add SMTP endpoint

2023-10-17 Thread Lukas Wagner
Ping - would be great to get some reviews on this to get this merged for the next release. On 9/18/23 13:14, Lukas Wagner wrote: This patch series adds support for a new notification endpoint type, smtp. As the name suggests, this new endpoint allows PVE to talk to SMTP server directly, without

Re: [pve-devel] [PATCH v2 storage 1/1] fix #254: iscsi: allow to configure multiple portals

2023-10-17 Thread Yuri Konotopov via pve-devel
--- Begin Message --- 17.10.2023 11:07, Fiona Ebner пишет: Am 16.10.23 um 18:08 schrieb Yuri Konotopov: It looks like the return value of iscsi_discovery is not used. So we can drop $res completely and regex too. It is used by the scan/iscsi API endpoint via scan_iscsi() in Storage.pm. Than

Re: [pve-devel] [PATCH v2 storage 1/1] fix #254: iscsi: allow to configure multiple portals

2023-10-17 Thread Fiona Ebner
Am 16.10.23 um 18:08 schrieb Yuri Konotopov: > > It looks like the return value of iscsi_discovery is not used. So we can > drop > > $res completely and regex too. > It is used by the scan/iscsi API endpoint via scan_iscsi() in Storage.pm. Best Regards, Fiona