[pve-devel] [PATCH v2 02/22] common: make InstallZfsOption members public

2024-02-21 Thread Aaron Lauterer
as they will be used directly by the auto installer Signed-off-by: Aaron Lauterer --- proxmox-installer-common/src/setup.rs | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proxmox-installer-common/src/setup.rs b/proxmox-installer-common/src/setup.rs index 03beb77.

[pve-devel] [PATCH v2 03/22] common: tui: use BTreeMap for predictable ordering

2024-02-21 Thread Aaron Lauterer
necessary for the disk selection and network interfaces maps to have tests with results that can be compared without much additional effort. Signed-off-by: Aaron Lauterer --- proxmox-installer-common/src/setup.rs | 8 proxmox-tui-installer/src/options.rs | 4 ++-- proxmox-tui-installer

[pve-devel] [PATCH v2 05/22] add auto-installer crate

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- Cargo.toml| 1 + Makefile | 1 + proxmox-auto-installer/Cargo.toml | 10 ++ proxmox-auto-installer/src/lib.rs | 0 4 files changed, 12 insertions(+) create mode 100644 proxmox-auto-installer/Cargo.toml

[pve-devel] [PATCH v2 08/22] auto-installer: add struct to hold udev info

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- proxmox-auto-installer/src/lib.rs | 1 + proxmox-auto-installer/src/udevinfo.rs | 9 + 2 files changed, 10 insertions(+) create mode 100644 proxmox-auto-installer/src/udevinfo.rs diff --git a/proxmox-auto-installer/src/lib.rs b/proxmox-auto-instal

[pve-devel] [PATCH v2 01/22] tui: common: move InstallConfig struct to common crate

2024-02-21 Thread Aaron Lauterer
It describes the data structure expected by the low-level-installer. We do this so we can use it in more than the TUI installer, for example the planned auto installer. Make the members public so we can easily implement a custom From method for each dependent crate. Signed-off-by: Aaron Lauterer

[pve-devel] [PATCH v2 10/22] auto-installer: add simple logging

2024-02-21 Thread Aaron Lauterer
Log to stdout and the file the binary needs to set up. This is a first variant. By using the log crate macros we can change that in the future without too much effort. Signed-off-by: Aaron Lauterer --- proxmox-auto-installer/Cargo.toml | 2 ++ proxmox-auto-installer/src/lib.rs | 1 + proxmox-

[pve-devel] [PATCH v2 14/22] unconfigured: add proxauto as option to start auto installer

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- unconfigured.sh | 8 1 file changed, 8 insertions(+) diff --git a/unconfigured.sh b/unconfigured.sh index cf7de83..f668be4 100755 --- a/unconfigured.sh +++ b/unconfigured.sh @@ -5,6 +5,7 @@ trap "err_reboot" ERR # NOTE: we nowadays get exec'd by the in

[pve-devel] [PATCH v2 22/22] control: update build depends for auto installer

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- debian/control | 10 ++ 1 file changed, 10 insertions(+) diff --git a/debian/control b/debian/control index 9057f59..72a5657 100644 --- a/debian/control +++ b/debian/control @@ -8,10 +8,20 @@ Build-Depends: cargo:native, libgtk3-perl,

[pve-devel] [PATCH v2 07/22] auto-installer: add answer file definition

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- changes since v1: * added default traits and switched to them proxmox-auto-installer/src/answer.rs | 132 +++ proxmox-auto-installer/src/lib.rs| 1 + 2 files changed, 133 insertions(+) create mode 100644 proxmox-auto-installer/src/

[pve-devel] [PATCH v2 19/22] auto-installer: helper: add subcommand to view indentifiers

2024-02-21 Thread Aaron Lauterer
It will collect the information from the current system and show the payload of identifiers that will be send. To avoid confusion, the subcommands for the device info and filter matching have been renamed. Signed-off-by: Aaron Lauterer --- .../src/bin/proxmox-autoinst-helper.rs| 54

[pve-devel] [PATCH v2 20/22] auto-installer: fetch: add http post utility module

2024-02-21 Thread Aaron Lauterer
It sends a http(s) POST request with the sysinfo as payload and expects an answer file in return. In order to handle non FQDN URLs (e.g. IP addresses) and self signed certificates, it can optionally take an SHA256 fingerprint of the certificate. This can of course also be used to pin a certificate

[pve-devel] [PATCH v2 13/22] auto-installer: add fetch answer binary

2024-02-21 Thread Aaron Lauterer
it is supposed to be run first and fetch an answer file. The initial implementation searches for a partition/filesystem called 'proxmoxinst' or 'PROXMOXINST' with an 'answer.toml' file in the root directory. Once it has an answer file, it will call the 'proxmox-auto-installer' and pipe in the con

[pve-devel] [PATCH v2 12/22] auto-installer: add auto-installer binary

2024-02-21 Thread Aaron Lauterer
It expects the contents of an answer file via stdin. It will then be parsed and the JSON for the low level installer is generated. It then calls the low level installer directly. The output of the installaton progress is kept rather simple for now. If configured in the answer file, commands will

[pve-devel] [PATCH v2 04/22] low-level: add dump-udev command

2024-02-21 Thread Aaron Lauterer
Fetches UDEV device properties prepended with 'E:' for NICs and disks. The result is stored in its own JSON file. This information is needed to filter for specific devices. Mainly for the auto-installer for now. Signed-off-by: Aaron Lauterer --- Proxmox/Makefile| 1 + Proxmox/Sys/U

[pve-devel] [PATCH v2 06/22] auto-installer: add dependencies

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- changes since v1: * toml switched from 0.5 to 0.7 proxmox-auto-installer/Cargo.toml | 4 1 file changed, 4 insertions(+) diff --git a/proxmox-auto-installer/Cargo.toml b/proxmox-auto-installer/Cargo.toml index 75cfb2c..67218dd 100644 --- a/proxmox-auto-in

[pve-devel] [PATCH v2 16/22] auto-installer: utils: make get_udev_index functions public

2024-02-21 Thread Aaron Lauterer
because we will need to access them directly in the future from a separate binary Signed-off-by: Aaron Lauterer --- proxmox-auto-installer/src/utils.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proxmox-auto-installer/src/utils.rs b/proxmox-auto-installer/src/utils.

[pve-devel] [PATCH v2 00/22] add automated/unattended installation

2024-02-21 Thread Aaron Lauterer
This patch series adds the possibility to do an automated / unattended installation of Proxmox VE. It assumes that the patch series to use JSON output on the low-level-installer is already applied [1]. The overall idea is that we will have a dedicated ISO for the unattended installation. It shoul

[pve-devel] [PATCH v2 15/22] auto-installer: use glob crate for pattern matching

2024-02-21 Thread Aaron Lauterer
Signed-off-by: Aaron Lauterer --- proxmox-auto-installer/Cargo.toml | 1 + proxmox-auto-installer/src/utils.rs | 48 +++-- 2 files changed, 19 insertions(+), 30 deletions(-) diff --git a/proxmox-auto-installer/Cargo.toml b/proxmox-auto-installer/Cargo.toml index 3dcb5

[pve-devel] [PATCH v2 21/22] auto-installer: fetch: add http plugin to fetch answer

2024-02-21 Thread Aaron Lauterer
This plugin will send a HTTP POST request with identifying sysinfo to fetch an answer file. The provided sysinfo can be used to identify the system and generate a matching answer file on demand. The URL to send the request to, can be defined in two ways. Via a custom DHCP option or a TXT record on

[pve-devel] [PATCH v2 18/22] auto-installer: fetch: add gathering of system identifiers and restructure code

2024-02-21 Thread Aaron Lauterer
They will be used as payload when POSTing a request for an answer file. The idea is, that with this information, it should be possible to identify the system and generate a matching answer file on the fly. Many of these properties can also be found on the machine or packaging of the machine and cou

[pve-devel] [PATCH v2 09/22] auto-installer: add utils

2024-02-21 Thread Aaron Lauterer
contains several utility structs and functions. For example: a simple pattern matcher that matches wildcards at the beginning or end of the filter. It currently uses a dedicated function (parse_answer) to generate the InstallConfig struct instead of a From implementation. This is because for now

[pve-devel] [PATCH v2 17/22] auto-installer: add proxmox-autoinst-helper tool

2024-02-21 Thread Aaron Lauterer
It can parse an answer file to check against syntax errors, test match filters against the current hardware and list properties of the current hardware to match against. Since this tool should be able to run outside of the installer environment, it does not rely on the device information provided

Re: [pve-devel] [PATCH v3 ceph master, ceph quincy-stable 8, pve-storage, pve-manager 00/13] Fix #4759: Configure Permissions for ceph-crash.service

2024-02-21 Thread Friedrich Weber
Quickly tested the patch series on my existing Ceph Quincy cluster, did not encounter major issues -- the keyring was created and the Ceph config was rewritten accordingly. After a restart of `ceph-crash`, it correctly posts crashes (produced with `kill -n11 $(pidof ceph-osd)`) again and does not w

[pve-devel] [PATCH v2 11/22] auto-installer: add tests for answer file parsing

2024-02-21 Thread Aaron Lauterer
By matching the resulting json to be passed to the low level installer against known good ones. The environment info was gathered from one of our AMD Epyc Rome test servers to have a realistic starting point. Signed-off-by: Aaron Lauterer --- proxmox-auto-installer/tests/parse-answer.rs | 102

Re: [pve-devel] [PATCH v2 20/22] auto-installer: fetch: add http post utility module

2024-02-21 Thread Christoph Heiss
On Wed, Feb 21, 2024 at 12:08:03PM +0100, Aaron Lauterer wrote: > It sends a http(s) POST request with the sysinfo as payload and expects > an answer file in return. > [..] > diff --git a/proxmox-auto-installer/src/fetch_plugins/utils/post.rs > b/proxmox-auto-installer/src/fetch_plugins/utils/post

[pve-devel] [PATCH qemu] add patch to fix deadlock with VirtIO block and iothread during QMP stop

2024-02-21 Thread Fiona Ebner
Backported from commit bfa36802d1 ("virtio-blk: avoid using ioeventfd state in irqfd conditional") because the rework/rename dataplane -> ioeventfd didn't happen yet. Reported in the community forum [0] and reproduced doing a backup loop to PBS with suspend mode with fio doing heavy IO in the gues

Re: [pve-devel] [PATCH v3 ceph master, ceph quincy-stable 8, pve-storage, pve-manager 00/13] Fix #4759: Configure Permissions for ceph-crash.service

2024-02-21 Thread Max Carrara
On 2/21/24 12:55, Friedrich Weber wrote: > Quickly tested the patch series on my existing Ceph Quincy cluster, did > not encounter major issues -- the keyring was created and the Ceph > config was rewritten accordingly. After a restart of `ceph-crash`, it > correctly posts crashes (produced with `k

Re: [pve-devel] [PATCH v2 00/22] add automated/unattended installation

2024-02-21 Thread Christoph Heiss
On Wed, Feb 21, 2024 at 12:07:43PM +0100, Aaron Lauterer wrote: > This patch series adds the possibility to do an automated / unattended > installation of Proxmox VE. > [..] Before anyone else stumbles into this while testing/reviewing, I've also just talked with Aaron over this: Basically, the bu

Re: [pve-devel] [PATCH v2 18/22] auto-installer: fetch: add gathering of system identifiers and restructure code

2024-02-21 Thread Christoph Heiss
On Wed, Feb 21, 2024 at 12:08:01PM +0100, Aaron Lauterer wrote: > They will be used as payload when POSTing a request for an answer file. The > idea is, that with this information, it should be possible to identify > the system and generate a matching answer file on the fly. > Many of these propert

Re: [pve-devel] [PATCH close #545, #5203: Allow bridges to have any valid interface name 1/2]

2024-02-21 Thread Stefan Hanreich
On 1/26/24 18:17, Jillian Morgan wrote: > Allow bridges to have any valid interface name > Repo: proxmox-widget-toolkit > > Allow the web UI to accept bridge interfaces with any valid interface name, > rather than being limited to the arbitrary "vmbr" prefix. > > Signed-off-by: Jillian Morgan

Re: [pve-devel] [PATCH close #545, #5203: Allow bridges to have any valid interface name 2/2]

2024-02-21 Thread Stefan Hanreich
On 1/26/24 18:17, Jillian Morgan wrote: > Detect bridge interface by bridge_ports attribute > Repo: pve-common > > Similar to other interface types, we can detect a bridge by the presense of > it's bridge_ports attribute rather than solely relying on the "vmbr" ifname > prefix heuristic. > > S

Re: [pve-devel] [PATCH close #545, #5203: Allow bridges to have any valid interface name 0/2]

2024-02-21 Thread Stefan Hanreich
Hi! Thanks for your patch! In general we like the idea of being able to choose arbitrary names for bridges - but there's some additional things to consider before we can apply those patches. We are currently using prefixes for other types of interfaces as well. With this patch series I could crea

Re: [pve-devel] [PATCH close #545, #5203: Allow bridges to have any valid interface name 2/2]

2024-02-21 Thread Stefan Hanreich
Hit send a bit too early, had another remark on this On 1/26/24 18:17, Jillian Morgan wrote: > Detect bridge interface by bridge_ports attribute > Repo: pve-common > > Similar to other interface types, we can detect a bridge by the presense of > it's bridge_ports attribute rather than solely rely

[pve-devel] [PATCH-SERIES v8 qemu-server, common] Prevent starting a 32-bit VM using a 64-bit OVMF BIOS

2024-02-21 Thread Filip Schauer
This patch series prevents starting a 32-bit VM using a 64-bit OVMF BIOS and makes the default value for 'kvm' during CPU hotplug consistent with the rest of the code. This is a breaking change for VMs with a different CPU architecture running on an x86_64 host. Changes since v7: * Rename is_nativ

[pve-devel] [PATCH qemu-server 1/5] cpu config: add helper to get the default CPU type

2024-02-21 Thread Filip Schauer
Signed-off-by: Filip Schauer --- PVE/QemuServer/CPUConfig.pm | 16 +++- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm index ca2946b..4be5262 100644 --- a/PVE/QemuServer/CPUConfig.pm +++ b/PVE/QemuServer/CPUConf

[pve-devel] [PATCH common 1/1] tools: add is_native_arch to compare the CPU architecture

2024-02-21 Thread Filip Schauer
Add an is_native_arch($arch) subroutine to compare a CPU architecture to the host CPU architecture. This is brought in from PVE::QemuServer. Signed-off-by: Filip Schauer --- src/PVE/Tools.pm | 6 ++ 1 file changed, 6 insertions(+) diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm index 766c8

[pve-devel] [PATCH qemu-server 2/5] prevent starting a 32-bit VM using a 64-bit OVMF BIOS

2024-02-21 Thread Filip Schauer
Instead of starting a VM with a 32-bit CPU type and a 64-bit OVMF image, throw an error before starting the VM telling the user that OVMF is not supported on 32-bit CPU types. To obtain a list of 32-bit CPU types, refer to the builtin_x86_defs in target/i386/cpu.c of QEMU. Exclude any entries that

[pve-devel] [PATCH qemu-server 4/5] Unify the default value for 'kvm'

2024-02-21 Thread Filip Schauer
Signed-off-by: Filip Schauer --- PVE/QemuServer.pm | 5 +++-- PVE/QemuServer/CPUConfig.pm | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index 7600939..6055d40 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServer.pm @@ -374

[pve-devel] [PATCH qemu-server 3/5] Move is_native from PVE::QemuServer to PVE::Tools

2024-02-21 Thread Filip Schauer
Move is_native from PVE::QemuServer to PVE::Tools and rename it to is_native_arch to be more descriptive. Signed-off-by: Filip Schauer --- PVE/QemuServer.pm | 13 - 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index 33c374c..7600

[pve-devel] [PATCH qemu-server 5/5] cpu config: die on hotplug of non x86_64 CPUs

2024-02-21 Thread Filip Schauer
When attempting a CPU hotplug on an architecture other than x86_64, die with a clean error instead of attempting a hotplug with a known non-working device command line. Also move the corresponding FIXME up to the error. Signed-off-by: Filip Schauer --- PVE/QemuServer/CPUConfig.pm | 4 +++- 1 fil

Re: [pve-devel] [PATCH common 1/1] tools: Add is_native sub to compare the CPU architecture

2024-02-21 Thread Filip Schauer
Patch v8 available: https://lists.proxmox.com/pipermail/pve-devel/2024-February/061899.html On 19/02/2024 15:46, Fiona Ebner wrote: Am 19.12.23 um 10:40 schrieb Filip Schauer: Add an is_native($arch) subroutine to compare a CPU architecture to the host CPU architecture. This is brought in from

Re: [pve-devel] [PATCH qemu-server 4/4] cpu config: Unify the default value for 'kvm'

2024-02-21 Thread Filip Schauer
On 19/02/2024 15:47, Fiona Ebner wrote: On an x86_64 host, for guests using a different architecture (i.e. aarch64), hot-plugging is already broken, because we try to hotplug a CPU of type "$cpu-x86_64-cpu,XYZ" which won't work anyways: vcpus: hotplug problem - VM 130 qmp command 'device_add' fa

Re: [pve-devel] [PATCH v2 18/22] auto-installer: fetch: add gathering of system identifiers and restructure code

2024-02-21 Thread Aaron Lauterer
On 2/21/24 15:09, Christoph Heiss wrote: While looking at this and reading the `dmidecode` manpage a bit, it looks like everything collected here is also available under /sys/devices/virtual/dmi/id as separate files, e.g. # ls /sys/devices/virtual/dmi/id/ bios_datechassis_as

[pve-devel] applied: [PATCH qemu] add patch to fix deadlock with VirtIO block and iothread during QMP stop

2024-02-21 Thread Thomas Lamprecht
Am 21/02/2024 um 14:01 schrieb Fiona Ebner: > Backported from commit bfa36802d1 ("virtio-blk: avoid using ioeventfd > state in irqfd conditional") because the rework/rename dataplane -> > ioeventfd didn't happen yet. > > Reported in the community forum [0] and reproduced doing a backup loop > to P