Re: [pve-devel] [PATCH container] fix #5194: delete environment variables set by pve
On January 22, 2024 11:12 am, Folke Gleumes wrote:
> proxmox-perl-rs set's SSL_CERT_{DIR,FILE}, which can break ssl in
> containers if their certificate store can't be found in the same spot.
> This patch explicitly unsets those variables before starting the
> container.
after a short talk with Wolfgang - this patch is probably an okay
stop-gap to fix the particular regression.
but it might be nice to switch to `--clear-env` for lxc-attach with
corresponding options for pct to either preserve the whole env, or
particular variables? might be 9.0 material since it is a semantic
change that possibly breaks scripted use cases that rely on env
variables to pass along things from host to whatever they run inside the
container.. we could introduce the options now though and also have a
`--keep-env` that is the default for 8.x, and flip it to default to
`--clear-env` with 9.0.
>
> Signed-off-by: Folke Gleumes
> ---
> src/PVE/CLI/pct.pm | 11 +++
> 1 file changed, 11 insertions(+)
>
> diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm
> index a0b9bce..53519e4 100755
> --- a/src/PVE/CLI/pct.pm
> +++ b/src/PVE/CLI/pct.pm
> @@ -143,6 +143,15 @@ __PACKAGE__->register_method ({
> exec(@$cmd);
> }});
>
> +sub clean_environment {
> +# These env variables are currently needed by PVE to work correctly with
> rust libraries,
> +# but can break ssl inside of containers.
> +# An explanation why they are needed and the code that sets them can be
> found here:
> +#
> https://git.proxmox.com/?p=proxmox-perl-rs.git;a=blob;f=common/pkg/Proxmox/Lib/SslProbe.pm
> +delete $ENV{SSL_CERT_FILE};
> +delete $ENV{SSL_CERT_DIR};
> +};
> +
> __PACKAGE__->register_method ({
> name => 'enter',
> path => 'enter',
> @@ -164,6 +173,7 @@ __PACKAGE__->register_method ({
> PVE::LXC::Config->load_config($vmid); # test if container exists on
> this node
> die "container '$vmid' not running!\n" if
> !PVE::LXC::check_running($vmid);
>
> + clean_environment();
> exec('lxc-attach', '-n', $vmid);
> }});
>
> @@ -189,6 +199,7 @@ __PACKAGE__->register_method ({
>
> die "missing command" if !@{$param->{'extra-args'}};
>
> + clean_environment();
> exec('lxc-attach', '-n', $vmid, '--', @{$param->{'extra-args'}});
> }});
>
> --
> 2.39.2
>
>
>
> ___
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 storage] lvm: improve warning in case vgs output contains unexpected lines
On 19/01/2024 12:31, Fiona Ebner wrote:
> Am 19.01.24 um 11:59 schrieb Fiona Ebner:
>> Am 18.01.24 um 12:11 schrieb Friedrich Weber:
>>> diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm
>>> index 4b951e7..5377823 100644
>>> --- a/src/PVE/Storage/LVMPlugin.pm
>>> +++ b/src/PVE/Storage/LVMPlugin.pm
>>> @@ -130,6 +130,11 @@ sub lvm_vgs {
>>>
>>> my ($name, $size, $free, $lvcount, $pvname, $pvsize, $pvfree) =
>>> split (':', $line);
>>>
>>> + if (!defined($size) || !defined($free) || !defined($lvcount)) {
>>> + warn "unexpected output from vgs: $line\n";
>>> + return;
>>> + }
>>> +
>>
>> Nit: maybe quote 'vgs' and/or say "command 'vgs'"?
Sounds good!
>> Please use log_warn() from PVE::RESTEnvironment for new warnings, so
>> they also show up in task logs.
>
> Sorry, I mean "show up more visibly", because they count towards the
> warning count shown in the task result.
Thanks, wasn't aware of this benefit of `log_warn`.
Will send a v3 with the two changes.
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH qemu] add patch to work around stuck guest IO with iothread and VirtIO block/SCSI
This essentially repeats commit 6b7c181 ("add patch to work around
stuck guest IO with iothread and VirtIO block/SCSI") with an added
fix for the SCSI event virtqueue, which requires special handling.
This is to avoid the issue [4] that made the revert 2a49e66 ("Revert
"add patch to work around stuck guest IO with iothread and VirtIO
block/SCSI"") necessary the first time around.
When using iothread, after commits
1665d9326f ("virtio-blk: implement BlockDevOps->drained_begin()")
766aa2de0f ("virtio-scsi: implement BlockDevOps->drained_begin()")
it can happen that polling gets stuck when draining. This would cause
IO in the guest to get completely stuck.
A workaround for users is stopping and resuming the vCPUs because that
would also stop and resume the dataplanes which would kick the host
notifiers.
This can happen with block jobs like backup and drive mirror as well
as with hotplug [2].
Reports in the community forum that might be about this issue[0][1]
and there is also one in the enterprise support channel.
As a workaround in the code, just re-enable notifications and kick the
virt queue after draining. Draining is already costly and rare, so no
need to worry about a performance penalty here. This was taken from
the following comment of a QEMU developer [3] (in my debugging,
I had already found re-enabling notification to work around the issue,
but also kicking the queue is more complete).
Take special care to attach the SCSI event virtqueue host notifier
with the _no_poll() variant like in virtio_scsi_dataplane_start().
This avoids the issue from the first attempted fix where the iothread
would suddenly loop with 100% CPU usage whenever some guest IO came in
[4]. This is necessary because of commit 38738f7dbb ("virtio-scsi:
don't waste CPU polling the event virtqueue"). See [5] for the
relevant discussion.
[0]: https://forum.proxmox.com/threads/137286/
[1]: https://forum.proxmox.com/threads/137536/
[2]: https://issues.redhat.com/browse/RHEL-3934
[3]:
https://issues.redhat.com/browse/RHEL-3934?focusedId=23562096&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-23562096
[4]: https://forum.proxmox.com/threads/138140/
[5]:
https://lore.kernel.org/qemu-devel/bfc7b20c-2144-46e9-acbc-e726276c5...@proxmox.com/
Signed-off-by: Fiona Ebner
---
...work-around-iothread-polling-getting.patch | 87 +++
debian/patches/series | 1 +
2 files changed, 88 insertions(+)
create mode 100644
debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
diff --git
a/debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
b/debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
new file mode 100644
index 000..a268eed
--- /dev/null
+++
b/debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
@@ -0,0 +1,87 @@
+From Mon Sep 17 00:00:00 2001
+From: Fiona Ebner
+Date: Tue, 23 Jan 2024 13:21:11 +0100
+Subject: [PATCH] virtio blk/scsi: work around iothread polling getting stuck
+ with drain
+
+When using iothread, after commits
+1665d9326f ("virtio-blk: implement BlockDevOps->drained_begin()")
+766aa2de0f ("virtio-scsi: implement BlockDevOps->drained_begin()")
+it can happen that polling gets stuck when draining. This would cause
+IO in the guest to get completely stuck.
+
+A workaround for users is stopping and resuming the vCPUs because that
+would also stop and resume the dataplanes which would kick the host
+notifiers.
+
+This can happen with block jobs like backup and drive mirror as well
+as with hotplug [2].
+
+Reports in the community forum that might be about this issue[0][1]
+and there is also one in the enterprise support channel.
+
+As a workaround in the code, just re-enable notifications and kick the
+virt queue after draining. Draining is already costly and rare, so no
+need to worry about a performance penalty here. This was taken from
+the following comment of a QEMU developer [3] (in my debugging,
+I had already found re-enabling notification to work around the issue,
+but also kicking the queue is more complete).
+
+Take special care to attach the SCSI event virtqueue host notifier
+with the _no_poll() variant like in virtio_scsi_dataplane_start().
+This avoids the issue from the first attempted fix where the iothread
+would suddenly loop with 100% CPU usage whenever some guest IO came in
+[4]. This is necessary because of commit 38738f7dbb ("virtio-scsi:
+don't waste CPU polling the event virtqueue"). See [5] for the
+relevant discussion.
+
+[0]: https://forum.proxmox.com/threads/137286/
+[1]: https://forum.proxmox.com/threads/137536/
+[2]: https://issues.redhat.com/browse/RHEL-3934
+[3]:
https://issues.redhat.com/browse/RHEL-3934?focusedId=23562096&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-23562096
+[4]: https://forum.proxmox.com/threads/
Re: [pve-devel] [PATCH qemu] add patch to work around stuck guest IO with iothread and VirtIO block/SCSI
Very good news Fiona
For quite some time I have been using patchlevel 5 of the pve-qemu
package (the one that has CPU overloads) because package 4-6 gives me
stuck storage problems, as you correctly describe in your post.
Very thanks !
Il 23/01/24 14:13, Fiona Ebner ha scritto:
This essentially repeats commit 6b7c181 ("add patch to work around
stuck guest IO with iothread and VirtIO block/SCSI") with an added
fix for the SCSI event virtqueue, which requires special handling.
This is to avoid the issue [4] that made the revert 2a49e66 ("Revert
"add patch to work around stuck guest IO with iothread and VirtIO
block/SCSI"") necessary the first time around.
When using iothread, after commits
1665d9326f ("virtio-blk: implement BlockDevOps->drained_begin()")
766aa2de0f ("virtio-scsi: implement BlockDevOps->drained_begin()")
it can happen that polling gets stuck when draining. This would cause
IO in the guest to get completely stuck.
A workaround for users is stopping and resuming the vCPUs because that
would also stop and resume the dataplanes which would kick the host
notifiers.
This can happen with block jobs like backup and drive mirror as well
as with hotplug [2].
Reports in the community forum that might be about this issue[0][1]
and there is also one in the enterprise support channel.
As a workaround in the code, just re-enable notifications and kick the
virt queue after draining. Draining is already costly and rare, so no
need to worry about a performance penalty here. This was taken from
the following comment of a QEMU developer [3] (in my debugging,
I had already found re-enabling notification to work around the issue,
but also kicking the queue is more complete).
Take special care to attach the SCSI event virtqueue host notifier
with the _no_poll() variant like in virtio_scsi_dataplane_start().
This avoids the issue from the first attempted fix where the iothread
would suddenly loop with 100% CPU usage whenever some guest IO came in
[4]. This is necessary because of commit 38738f7dbb ("virtio-scsi:
don't waste CPU polling the event virtqueue"). See [5] for the
relevant discussion.
[0]: https://forum.proxmox.com/threads/137286/
[1]: https://forum.proxmox.com/threads/137536/
[2]: https://issues.redhat.com/browse/RHEL-3934
[3]:
https://issues.redhat.com/browse/RHEL-3934?focusedId=23562096&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-23562096
[4]: https://forum.proxmox.com/threads/138140/
[5]:
https://lore.kernel.org/qemu-devel/bfc7b20c-2144-46e9-acbc-e726276c5...@proxmox.com/
Signed-off-by: Fiona Ebner
---
...work-around-iothread-polling-getting.patch | 87 +++
debian/patches/series | 1 +
2 files changed, 88 insertions(+)
create mode 100644
debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
diff --git
a/debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
b/debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
new file mode 100644
index 000..a268eed
--- /dev/null
+++
b/debian/patches/pve/0046-virtio-blk-scsi-work-around-iothread-polling-getting.patch
@@ -0,0 +1,87 @@
+From Mon Sep 17 00:00:00 2001
+From: Fiona Ebner
+Date: Tue, 23 Jan 2024 13:21:11 +0100
+Subject: [PATCH] virtio blk/scsi: work around iothread polling getting stuck
+ with drain
+
+When using iothread, after commits
+1665d9326f ("virtio-blk: implement BlockDevOps->drained_begin()")
+766aa2de0f ("virtio-scsi: implement BlockDevOps->drained_begin()")
+it can happen that polling gets stuck when draining. This would cause
+IO in the guest to get completely stuck.
+
+A workaround for users is stopping and resuming the vCPUs because that
+would also stop and resume the dataplanes which would kick the host
+notifiers.
+
+This can happen with block jobs like backup and drive mirror as well
+as with hotplug [2].
+
+Reports in the community forum that might be about this issue[0][1]
+and there is also one in the enterprise support channel.
+
+As a workaround in the code, just re-enable notifications and kick the
+virt queue after draining. Draining is already costly and rare, so no
+need to worry about a performance penalty here. This was taken from
+the following comment of a QEMU developer [3] (in my debugging,
+I had already found re-enabling notification to work around the issue,
+but also kicking the queue is more complete).
+
+Take special care to attach the SCSI event virtqueue host notifier
+with the _no_poll() variant like in virtio_scsi_dataplane_start().
+This avoids the issue from the first attempted fix where the iothread
+would suddenly loop with 100% CPU usage whenever some guest IO came in
+[4]. This is necessary because of commit 38738f7dbb ("virtio-scsi:
+don't waste CPU polling the event virtqueue"). See [5] for the
+relevant discussion.
+
+[0]: https://forum.proxmox.com/threads/137286
[pve-devel] [PATCH v1 installer 04/18] Makefile: fix handling of multiple usr_bin files
Otherwise the build will fail once we define more than one USR_BIN file. Signed-off-by: Aaron Lauterer --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 601c836..f0c361b 100644 --- a/Makefile +++ b/Makefile @@ -104,7 +104,7 @@ install: $(INSTALLER_SOURCES) $(CARGO_COMPILEDIR)/proxmox-tui-installer install -D -m 755 unconfigured.sh $(DESTDIR)/sbin/unconfigured.sh install -D -m 755 proxinstall $(DESTDIR)/usr/bin/proxinstall install -D -m 755 proxmox-low-level-installer $(DESTDIR)/$(BINDIR)/proxmox-low-level-installer - $(foreach i,$(USR_BIN), install -m755 $(CARGO_COMPILEDIR)/$(i) $(DESTDIR)$(BINDIR)/) + $(foreach i,$(USR_BIN), install -m755 $(CARGO_COMPILEDIR)/$(i) $(DESTDIR)$(BINDIR)/ ;) install -D -m 755 checktime $(DESTDIR)/usr/bin/checktime install -D -m 644 xinitrc $(DESTDIR)/.xinitrc install -D -m 755 spice-vdagent.sh $(DESTDIR)/.spice-vdagent.sh -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 03/18] common: tui: use BTreeMap for predictable ordering
necessary for the disk selection and network interfaces maps to have
tests with results that can be compared without much additional effort.
Signed-off-by: Aaron Lauterer
---
proxmox-installer-common/src/setup.rs | 8
proxmox-tui-installer/src/options.rs | 4 ++--
proxmox-tui-installer/src/setup.rs| 4 ++--
3 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/proxmox-installer-common/src/setup.rs
b/proxmox-installer-common/src/setup.rs
index 1bc4cb7..8432a2c 100644
--- a/proxmox-installer-common/src/setup.rs
+++ b/proxmox-installer-common/src/setup.rs
@@ -1,6 +1,6 @@
use std::{
cmp,
-collections::HashMap,
+collections::{BTreeMap, HashMap},
fmt,
fs::File,
io::{self, BufReader},
@@ -297,7 +297,7 @@ pub struct NetworkInfo {
/// Maps devices to their configuration, if it has a usable configuration.
/// (Contains no entries for devices with only link-local addresses.)
#[serde(default)]
-pub interfaces: HashMap,
+pub interfaces: BTreeMap,
/// The hostname of this machine, if set by the DHCP server.
pub hostname: Option,
@@ -416,8 +416,8 @@ pub struct InstallConfig {
skip_serializing_if = "Option::is_none"
)]
pub target_hd: Option,
-#[serde(skip_serializing_if = "HashMap::is_empty")]
-pub disk_selection: HashMap,
+#[serde(skip_serializing_if = "BTreeMap::is_empty")]
+pub disk_selection: BTreeMap,
pub country: String,
pub timezone: String,
diff --git a/proxmox-tui-installer/src/options.rs
b/proxmox-tui-installer/src/options.rs
index 094a430..73fbf2a 100644
--- a/proxmox-tui-installer/src/options.rs
+++ b/proxmox-tui-installer/src/options.rs
@@ -76,7 +76,7 @@ mod tests {
utils::{CidrAddress, Fqdn},
};
use std::net::{IpAddr, Ipv4Addr};
-use std::{collections::HashMap, path::PathBuf};
+use std::{collections::BTreeMap, path::PathBuf};
fn dummy_setup_info() -> SetupInfo {
SetupInfo {
@@ -99,7 +99,7 @@ mod tests {
fn network_options_from_setup_network_info() {
let setup = dummy_setup_info();
-let mut interfaces = HashMap::new();
+let mut interfaces = BTreeMap::new();
interfaces.insert(
"eth0".to_owned(),
Interface {
diff --git a/proxmox-tui-installer/src/setup.rs
b/proxmox-tui-installer/src/setup.rs
index e816c12..248f86e 100644
--- a/proxmox-tui-installer/src/setup.rs
+++ b/proxmox-tui-installer/src/setup.rs
@@ -1,4 +1,4 @@
-use std::collections::HashMap;
+use std::collections::BTreeMap;
use crate::options::InstallerOptions;
use proxmox_installer_common::{
@@ -19,7 +19,7 @@ impl From for InstallConfig {
maxvz: None,
zfs_opts: None,
target_hd: None,
-disk_selection: HashMap::new(),
+disk_selection: BTreeMap::new(),
country: options.timezone.country,
timezone: options.timezone.timezone,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 09/18] auto-installer: add struct to hold udev info
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/lib.rs | 1 +
proxmox-auto-installer/src/udevinfo.rs | 9 +
2 files changed, 10 insertions(+)
create mode 100644 proxmox-auto-installer/src/udevinfo.rs
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index 7813b98..8cda416 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -1 +1,2 @@
pub mod answer;
+pub mod udevinfo;
diff --git a/proxmox-auto-installer/src/udevinfo.rs
b/proxmox-auto-installer/src/udevinfo.rs
new file mode 100644
index 000..a6b61b5
--- /dev/null
+++ b/proxmox-auto-installer/src/udevinfo.rs
@@ -0,0 +1,9 @@
+use serde::Deserialize;
+use std::collections::BTreeMap;
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct UdevInfo {
+// use BTreeMap to have keys sorted
+pub disks: BTreeMap>,
+pub nics: BTreeMap>,
+}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 06/18] add auto-installer crate
Signed-off-by: Aaron Lauterer --- Cargo.toml| 1 + Makefile | 1 + proxmox-auto-installer/Cargo.toml | 10 ++ proxmox-auto-installer/src/lib.rs | 0 4 files changed, 12 insertions(+) create mode 100644 proxmox-auto-installer/Cargo.toml create mode 100644 proxmox-auto-installer/src/lib.rs diff --git a/Cargo.toml b/Cargo.toml index c1bd578..7017ac5 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,6 @@ [workspace] members = [ +"proxmox-auto-installer", "proxmox-installer-common", "proxmox-tui-installer", ] diff --git a/Makefile b/Makefile index f0c361b..57bd7ae 100644 --- a/Makefile +++ b/Makefile @@ -42,6 +42,7 @@ $(BUILDDIR): policy-disable-rc.d \ proxinstall \ proxmox-low-level-installer \ + proxmox-auto-installer/ \ proxmox-tui-installer/ \ proxmox-installer-common/ \ spice-vdagent.sh \ diff --git a/proxmox-auto-installer/Cargo.toml b/proxmox-auto-installer/Cargo.toml new file mode 100644 index 000..75cfb2c --- /dev/null +++ b/proxmox-auto-installer/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "proxmox-auto-installer" +version = "0.1.0" +edition = "2021" +authors = [ "Aaron Lauterer " ] +license = "AGPL-3" +exclude = [ "build", "debian" ] +homepage = "https://www.proxmox.com"; + +[dependencies] diff --git a/proxmox-auto-installer/src/lib.rs b/proxmox-auto-installer/src/lib.rs new file mode 100644 index 000..e69de29 -- 2.39.2 ___ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer/docs 00/18] add automated/unattended installation
This patch series adds the possibility to do an automated / unattended
installation of Proxmox VE.
It assumes that the patch series to use JSON output on the
low-level-installer is already applied [1].
The overall idea is that we will have a dedicated ISO for the unattended
installation. It should be configured in such a way that it will start
the installation without any user interaction.
Though the integration in the installation environmend isn't part of
this patch series.
The information for the installer that is usually gathered interactively
from the user is provided via an `answer.toml` file.
The answer file allows to select disks and the network card via filters.
The installer also allows to run custom commands pre and post
installation. This should give users plenty of possibilities to either
further customize/prepare the installation or integrate it into a larger
automated installation setup.
For example, one could issue HTTP requests to signal the status and
progress of the installation.
The install environment needs to call the 'proxmox-fetch-answer' binary.
It tries to find the answer file and once found, will start the
'proxmox-auto-installer' binary and pass the contents to it via stdin.
The auto-installer then parses the answer file and determines what
parameters need to be passed to the low-level installer. For example,
which disks and NIC to use, network IP settings and so forth.
The current status reporting of the actual installation is kept rather
simple.
Both binaries log into the /tmp/ directory.
There is a third binary, the 'proxmox-installer-filter'. It is meant as
a pure utility for users to make it easier to see what properties they
can write filters against and to test the filters.
The fetch-answer binary is currently searching for a
partition/file-system labeled 'proxmoxinst' in lower or uppercase. It
can be located on an additioan USB flash drive, or maybe on the install
medium itself if it is possible to write to it.
We do have some ideas for additional steps to fetch an answer file. The
main one is that we could download the answer file from a URL. Ideally
we would send unique properties along with the request (MAC addresses,
serial numbers, ...) so that it is possible to have a script on the
receiving side that can then generate the answer file dynamically.
The big question is, where the URL comes from, for which we have also
some ideas:
* custom DHCP options
* kernel cmdline (might be an option with PXE boot)
* TXT DNS record in a predefined subdomain of the search domain received
via DHCP, basically a 'dig TXT proxmoxinst.{search domain}'.
* We should also make it possible to provide an SSL fingerprint in a
similar manner in case the listening server is not trusted out of the
box.
Other plans / ideas for the future:
* add option to define remote SSH access (password and,or public key).
This could make remote debugging in case of problems easier
Regarding the patch series itself:
The first patches are needed to move some code into the common crate and
make structs/functions already in the common crate accessible.
I did split up the individual parts of the auto installer into their own
patches as much as possible, and (hopefully) in the order they depend on
each other.
Areas that can be improved/extended:
* Testing possibility integrated in the Makefile
* Documentation: explain process, additional examples for answer.toml
[0] https://lists.proxmox.com/pipermail/pve-devel/2023-September/059020.html
[1] https://lists.proxmox.com/pipermail/pve-devel/2023-December/060961.html
installer: Aaron Lauterer (17):
tui: common: move InstallConfig struct to common crate
common: make InstallZfsOption members public
common: tui: use BTreeMap for predictable ordering
Makefile: fix handling of multiple usr_bin files
low-level: add dump-udev command
add auto-installer crate
auto-installer: add dependencies
auto-installer: add answer file definition
auto-installer: add struct to hold udev info
auto-installer: add utils
auto-installer: add simple logging
auto-installer: add tests for answer file parsing
auto-installer: add auto-installer binary
auto-installer: add fetch answer binary
auto-installer: use glob crate for pattern matching
auto-installer: utils: make get_udev_index functions public
auto-installer: add proxmox-installer-filter helper tool
docs: Aaron Lauterer (1):
installation: add unattended documentation
pve-installation.adoc | 267 ++
1 file changed, 267 insertions(+)
Cargo.toml| 1 +
Makefile | 9 +-
Proxmox/Makefile | 1 +
Proxmox/Sys/Udev.pm | 54 ++
proxmox-auto-installer/Cargo.toml | 19 +
proxmox-auto-installer/src/answer.rs | 148 ++
.../src/bin/proxmox-auto-installer.rs | 192
.../src/b
[pve-devel] [PATCH v1 installer 07/18] auto-installer: add dependencies
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/Cargo.toml | 4
1 file changed, 4 insertions(+)
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 75cfb2c..211c605 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -8,3 +8,7 @@ exclude = [ "build", "debian" ]
homepage = "https://www.proxmox.com";
[dependencies]
+proxmox-installer-common = { path = "../proxmox-installer-common" }
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+toml = "0.5.11"
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 16/18] auto-installer: utils: make get_udev_index functions public
because we will need to access them directly in the future from a
separate binary
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/utils.rs | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/proxmox-auto-installer/src/utils.rs
b/proxmox-auto-installer/src/utils.rs
index 5990e93..283dbdc 100644
--- a/proxmox-auto-installer/src/utils.rs
+++ b/proxmox-auto-installer/src/utils.rs
@@ -79,7 +79,7 @@ pub fn get_network_settings(
Ok(network_options)
}
-fn get_single_udev_index(
+pub fn get_single_udev_index(
filter: BTreeMap,
udev_list: &BTreeMap>,
) -> Result {
@@ -104,7 +104,7 @@ fn get_single_udev_index(
Ok(dev_index.unwrap())
}
-fn get_matched_udev_indexes(
+pub fn get_matched_udev_indexes(
filter: BTreeMap,
udev_list: &BTreeMap>,
match_all: bool,
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 05/18] low-level: add dump-udev command
Fetches UDEV device properties prepended with 'E:' for NICs and disks.
The result is stored in its own JSON file.
This information is needed to filter for specific devices. Mainly for
the auto-installer for now.
Signed-off-by: Aaron Lauterer
---
Proxmox/Makefile| 1 +
Proxmox/Sys/Udev.pm | 54 +
proxmox-low-level-installer | 13 +
3 files changed, 68 insertions(+)
create mode 100644 Proxmox/Sys/Udev.pm
diff --git a/Proxmox/Makefile b/Proxmox/Makefile
index d49da80..9561d9b 100644
--- a/Proxmox/Makefile
+++ b/Proxmox/Makefile
@@ -16,6 +16,7 @@ PERL_MODULES=\
Sys/Command.pm \
Sys/File.pm \
Sys/Net.pm \
+Sys/Udev.pm \
UI.pm \
UI/Base.pm \
UI/Gtk3.pm \
diff --git a/Proxmox/Sys/Udev.pm b/Proxmox/Sys/Udev.pm
new file mode 100644
index 000..69d674f
--- /dev/null
+++ b/Proxmox/Sys/Udev.pm
@@ -0,0 +1,54 @@
+package Proxmox::Sys::Udev;
+
+use strict;
+use warnings;
+
+use base qw(Exporter);
+our @EXPORT_OK = qw(disk_details);
+
+my $udev_regex = '^E: ([A-Z_]*)=(.*)$';
+
+my sub fetch_udevadm_info {
+my ($path) = @_;
+
+my $info = `udevadm info --path $path --query all`;
+if (!$info) {
+ warn "no details found for device '${path}'\n";
+ next;
+}
+my $details = {};
+for my $line (split('\n', $info)) {
+ if ($line =~ m/$udev_regex/) {
+ $details->{$1} = $2;
+ }
+}
+return $details;
+}
+
+# return hash of E: properties returned by udevadm
+sub disk_details {
+my $result = {};
+for my $data (@{Proxmox::Sys::Block::get_cached_disks()}) {
+ my $index = @$data[0];
+ my $bd = @$data[5];
+ $result->{$index} = fetch_udevadm_info($bd);
+}
+return $result;
+}
+
+
+sub nic_details {
+my $nic_path = "/sys/class/net";
+my $result = {};
+
+my $nics = Proxmox::Sys::Net::get_ip_config()->{ifaces};
+
+for my $index (keys %$nics) {
+ my $name = $nics->{$index}->{name};
+ my $nic = "${nic_path}/${name}";
+ $result->{$name} = fetch_udevadm_info($nic);
+}
+return $result;
+}
+
+1;
diff --git a/proxmox-low-level-installer b/proxmox-low-level-installer
index d127a40..531251d 100755
--- a/proxmox-low-level-installer
+++ b/proxmox-low-level-installer
@@ -21,6 +21,7 @@ use Time::HiRes qw(usleep);
use Proxmox::Install::ISOEnv;
use Proxmox::Install::RunEnv;
+use Proxmox::Sys::Udev;
use Proxmox::Sys::File qw(file_write_all);
@@ -31,6 +32,7 @@ use Proxmox::UI;
my $commands = {
'dump-env' => 'Dump the current ISO and Hardware environment to base the
installer UI on.',
+'dump-udev' => 'Dump disk and network device info. Used for the auto
installation.',
'start-session' => 'Start an installation session, with command and result
transmitted via stdin/out',
'start-session-test' => 'Start an installation TEST session, with command
and result transmitted via stdin/out',
'help' => 'Output this usage help.',
@@ -114,6 +116,17 @@ if ($cmd eq 'dump-env') {
my $run_env = Proxmox::Install::RunEnv::query_installation_environment();
my $run_env_serialized = to_json($run_env, {canonical => 1, utf8 => 1})
."\n";
file_write_all($run_env_file, $run_env_serialized);
+} elsif ($cmd eq 'dump-udev') {
+my $out_dir = $env->{locations}->{run};
+make_path($out_dir);
+die "failed to create output directory '$out_dir'\n" if !-d $out_dir;
+
+my $output = {};
+$output->{disks} = Proxmox::Sys::Udev::disk_details();
+$output->{nics} = Proxmox::Sys::Udev::nic_details();
+
+my $output_serialized = to_json($output, {canonical => 1, utf8 => 1})
."\n";
+file_write_all("$out_dir/run-env-udev.json", $output_serialized);
} elsif ($cmd eq 'start-session') {
Proxmox::UI::init_stdio({}, $env);
read_and_merge_config();
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 11/18] auto-installer: add simple logging
Log to stdout and the file the binary needs to set up.
This is a first variant. By using the log crate macros we can change
that in the future without too much effort.
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/Cargo.toml | 2 ++
proxmox-auto-installer/src/lib.rs | 1 +
proxmox-auto-installer/src/log.rs | 38 +++
3 files changed, 41 insertions(+)
create mode 100644 proxmox-auto-installer/src/log.rs
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 211c605..078a333 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -8,7 +8,9 @@ exclude = [ "build", "debian" ]
homepage = "https://www.proxmox.com";
[dependencies]
+anyhow = "1.0"
proxmox-installer-common = { path = "../proxmox-installer-common" }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
toml = "0.5.11"
+log = "0.4.20"
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index 72884c1..6636cc7 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -1,3 +1,4 @@
pub mod answer;
+pub mod log;
pub mod udevinfo;
pub mod utils;
diff --git a/proxmox-auto-installer/src/log.rs
b/proxmox-auto-installer/src/log.rs
new file mode 100644
index 000..d52912a
--- /dev/null
+++ b/proxmox-auto-installer/src/log.rs
@@ -0,0 +1,38 @@
+use anyhow::{bail, Result};
+use log::{Level, Metadata, Record};
+use std::{fs::File, io::Write, sync::Mutex, sync::OnceLock};
+
+pub struct AutoInstLogger;
+static LOGFILE: OnceLock> = OnceLock::new();
+
+impl AutoInstLogger {
+pub fn init(path: &str) -> Result<()> {
+let f = File::create(path)?;
+if LOGFILE.set(Mutex::new(f)).is_err() {
+bail!("Cannot set LOGFILE")
+}
+Ok(())
+}
+}
+
+impl log::Log for AutoInstLogger {
+fn enabled(&self, metadata: &Metadata) -> bool {
+metadata.level() <= Level::Info
+}
+
+/// Logs to stdout without log level and into log file including log level
+fn log(&self, record: &Record) {
+if self.enabled(record.metadata()) {
+println!("{}", record.args());
+let mut file = LOGFILE
+.get()
+.expect("could not get LOGFILE")
+.lock()
+.expect("could not get mutex for LOGFILE");
+file.write_all(format!("{} - {}\n", record.level(),
record.args()).as_bytes())
+.expect("could not write to LOGFILE");
+}
+}
+
+fn flush(&self) {}
+}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 13/18] auto-installer: add auto-installer binary
It expects the contents of an answer file via stdin. It will then be
parsed and the JSON for the low level installer is generated.
It then calls the low level installer directly.
The output of the installaton progress is kept rather simple for now.
If configured in the answer file, commands will be run pre and post the
low level installer.
It also logs everything to the logfile, currently
'/tmp/auto_installer.log'.
Signed-off-by: Aaron Lauterer
---
Makefile | 4 +-
.../src/bin/proxmox-auto-installer.rs | 192 ++
2 files changed, 195 insertions(+), 1 deletion(-)
create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
diff --git a/Makefile b/Makefile
index 57bd7ae..9c933d9 100644
--- a/Makefile
+++ b/Makefile
@@ -18,7 +18,9 @@ INSTALLER_SOURCES=$(shell git ls-files) country.dat
PREFIX = /usr
BINDIR = $(PREFIX)/bin
-USR_BIN := proxmox-tui-installer
+USR_BIN := \
+ proxmox-tui-installer\
+ proxmox-auto-installer
COMPILED_BINS := \
$(addprefix $(CARGO_COMPILEDIR)/,$(USR_BIN))
diff --git a/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
b/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
new file mode 100644
index 000..8ede670
--- /dev/null
+++ b/proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
@@ -0,0 +1,192 @@
+use anyhow::{anyhow, bail, Result};
+use log::{error, info, LevelFilter};
+use std::{
+env,
+io::{BufRead, BufReader, Write},
+path::PathBuf,
+process::ExitCode,
+};
+
+use proxmox_installer_common::setup::{
+installer_setup, read_json, spawn_low_level_installer, LocaleInfo,
RuntimeInfo, SetupInfo,
+};
+
+use proxmox_auto_installer::{
+answer::Answer,
+log::AutoInstLogger,
+udevinfo::UdevInfo,
+utils,
+utils::{parse_answer, LowLevelMessage},
+};
+
+static LOGGER: AutoInstLogger = AutoInstLogger;
+
+pub fn init_log() -> Result<()> {
+AutoInstLogger::init("/tmp/auto_installer.log")?;
+log::set_logger(&LOGGER)
+.map(|()| log::set_max_level(LevelFilter::Info))
+.map_err(|err| anyhow!(err))
+}
+
+fn auto_installer_setup(in_test_mode: bool) -> Result<(Answer, UdevInfo)> {
+let base_path = if in_test_mode { "./testdir" } else { "/" };
+let mut path = PathBuf::from(base_path);
+
+path.push("run");
+path.push("proxmox-installer");
+
+let udev_info: UdevInfo = {
+let mut path = path.clone();
+path.push("run-env-udev.json");
+
+read_json(&path).map_err(|err| anyhow!("Failed to retrieve udev info
details: {err}"))?
+};
+
+let mut buffer = String::new();
+let lines = std::io::stdin().lock().lines();
+for line in lines {
+buffer.push_str(&line.unwrap());
+buffer.push('\n');
+}
+
+let answer: Answer =
+toml::from_str(&buffer).map_err(|err| anyhow!("Failed parsing answer
file: {err}"))?;
+
+Ok((answer, udev_info))
+}
+
+fn main() -> ExitCode {
+if let Err(err) = init_log() {
+panic!("could not initilize logging: {}", err);
+}
+
+let in_test_mode = match env::args().nth(1).as_deref() {
+Some("-t") => true,
+// Always force the test directory in debug builds
+_ => cfg!(debug_assertions),
+};
+info!("Starting auto installer");
+
+let (setup_info, locales, runtime_info) = match
installer_setup(in_test_mode) {
+Ok(result) => result,
+Err(err) => {
+error!("Installer setup error: {err}");
+return ExitCode::FAILURE;
+}
+};
+
+let (answer, udevadm_info) = match auto_installer_setup(in_test_mode) {
+Ok(result) => result,
+Err(err) => {
+error!("Autoinstaller setup error: {err}");
+return ExitCode::FAILURE;
+}
+};
+
+match utils::run_cmds("Pre", &answer.global.pre_command) {
+Ok(_) => (),
+Err(err) => {
+error!("Error when running Pre-Commands: {}", err);
+return exit_failure(answer.global.reboot_on_error);
+}
+};
+match run_installation(&answer, &locales, &runtime_info, &udevadm_info,
&setup_info) {
+Ok(_) => info!("Installation done."),
+Err(err) => {
+error!("Installation failed: {err}");
+return exit_failure(answer.global.reboot_on_error);
+}
+}
+match utils::run_cmds("Post", &answer.global.post_command) {
+Ok(_) => (),
+Err(err) => {
+error!("Error when running Post-Commands: {}", err);
+return exit_failure(answer.global.reboot_on_error);
+}
+};
+ExitCode::SUCCESS
+}
+
+/// When we exit with a failure, the installer will not automatically reboot.
+/// Default value for reboot_on_error is false
+fn exit_failure(reboot_on_error: Option) -> ExitCode {
+if let Some(true) = reboot_on_error {
+ExitCode::SUCCESS
+} else {
+ExitCode::FAILURE
+}
+}
+
+fn run_insta
[pve-devel] [PATCH v1 installer 14/18] auto-installer: add fetch answer binary
it is supposed to be run first and fetch an answer file.
The initial implementation searches for a partition/filesystem called
'proxmoxinst' or 'PROXMOXINST' with an 'answer.toml' file in the root
directory.
Once it has an answer file, it will call the 'proxmox-auto-installer'
and pipe in the contents via stdin.
Signed-off-by: Aaron Lauterer
---
Makefile | 1 +
.../src/bin/proxmox-fetch-answer.rs | 73 +
.../src/fetch_plugins/mod.rs | 1 +
.../src/fetch_plugins/partition.rs| 102 ++
proxmox-auto-installer/src/lib.rs | 1 +
5 files changed, 178 insertions(+)
create mode 100644 proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
create mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs
create mode 100644 proxmox-auto-installer/src/fetch_plugins/partition.rs
diff --git a/Makefile b/Makefile
index 9c933d9..b724789 100644
--- a/Makefile
+++ b/Makefile
@@ -20,6 +20,7 @@ PREFIX = /usr
BINDIR = $(PREFIX)/bin
USR_BIN := \
proxmox-tui-installer\
+ proxmox-fetch-answer\
proxmox-auto-installer
COMPILED_BINS := \
diff --git a/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
new file mode 100644
index 000..baf2bd2
--- /dev/null
+++ b/proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
@@ -0,0 +1,73 @@
+use anyhow::{anyhow, bail, Result};
+use log::{error, info, LevelFilter};
+use proxmox_auto_installer::fetch_plugins::partition::FetchFromPartition;
+use proxmox_auto_installer::log::AutoInstLogger;
+use std::io::Write;
+use std::process::{Command, ExitCode, Stdio};
+
+static LOGGER: AutoInstLogger = AutoInstLogger;
+
+pub fn init_log() -> Result<()> {
+AutoInstLogger::init("/tmp/fetch_answer.log")?;
+log::set_logger(&LOGGER)
+.map(|()| log::set_max_level(LevelFilter::Info))
+.map_err(|err| anyhow!(err))
+}
+
+fn fetch_answer() -> Result {
+info!("Checking for partition");
+match FetchFromPartition::get_answer() {
+Ok(answer) => return Ok(answer),
+Err(err) => info!("Fetching answer file from partition failed: {}",
err),
+}
+// TODO: add more options to get an answer file, e.g. download from url
where url could be
+// fetched via txt records on predefined subdomain, kernel param, dhcp
option, ...
+
+bail!("Could not find any answer file!")
+}
+
+fn main() -> ExitCode {
+if let Err(err) = init_log() {
+panic!("could not initilize logging: {err}");
+}
+
+info!("Fetching answer file");
+let answer = match fetch_answer() {
+Ok(answer) => answer,
+Err(err) => {
+error!("Aborting: {}", err);
+return ExitCode::FAILURE;
+}
+};
+
+let mut child = match Command::new("/usr/bin/proxmox-auto-installer")
+.stdout(Stdio::inherit())
+.stdin(Stdio::piped())
+.stderr(Stdio::null())
+.spawn()
+{
+Ok(child) => child,
+Err(err) => panic!("Failed to start automatic installation: {}", err),
+};
+
+let mut stdin = child.stdin.take().expect("Failed to open stdin");
+std::thread::spawn(move || {
+stdin
+.write_all(answer.as_bytes())
+.expect("Failed to write to stdin");
+});
+
+match child.wait() {
+Ok(status) => {
+if status.success() {
+ExitCode::SUCCESS
+} else {
+ExitCode::FAILURE // Will be trapped
+}
+}
+Err(err) => {
+error!("Auto installer exited: {err}");
+ExitCode::FAILURE
+}
+}
+}
diff --git a/proxmox-auto-installer/src/fetch_plugins/mod.rs
b/proxmox-auto-installer/src/fetch_plugins/mod.rs
new file mode 100644
index 000..44399e6
--- /dev/null
+++ b/proxmox-auto-installer/src/fetch_plugins/mod.rs
@@ -0,0 +1 @@
+pub mod partition;
diff --git a/proxmox-auto-installer/src/fetch_plugins/partition.rs
b/proxmox-auto-installer/src/fetch_plugins/partition.rs
new file mode 100644
index 000..0552ddd
--- /dev/null
+++ b/proxmox-auto-installer/src/fetch_plugins/partition.rs
@@ -0,0 +1,102 @@
+use anyhow::{bail, Result};
+use log::{info, warn};
+use std::fs::read_to_string;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+static ANSWER_FILE: &str = "answer.toml";
+static ANSWER_MP: &str = "/mnt/answer";
+static PARTLABEL: &str = "proxmoxinst";
+static SEARCH_PATH: &str = "/dev/disk/by-label";
+
+pub struct FetchFromPartition;
+
+impl FetchFromPartition {
+/// Returns the contents of the answer file
+pub fn get_answer() -> Result {
+let part_path = Self::scan_partlabels()?;
+Self::mount_part(part_path)?;
+Self::get_answer_file()
+}
+
+/// Searches for upper and lower case existence of the PARTLABEL in the
SEARCH_PATH
+fn scan_partlabels() -> Result {
+let p
[pve-devel] [PATCH v1 docs 18/18] installation: add unattended documentation
Signed-off-by: Aaron Lauterer --- Once we have defined the process on how it can be started, what the ISO is called and so forth, we can include that in the documentation. We should also add an example section to showcase the possibilities better. Maybe also explain how the post/pre commands can be used. Maybe even how to chroot into the installation to add further customizations. For the latter, a helper chroot tool, like the arch-chroot might be useful to have. pve-installation.adoc | 267 ++ 1 file changed, 267 insertions(+) diff --git a/pve-installation.adoc b/pve-installation.adoc index 1e909e2..5d6d887 100644 --- a/pve-installation.adoc +++ b/pve-installation.adoc @@ -300,6 +300,273 @@ following command: # zpool add log +[[installation_auto]] +Unattended Installation +--- + +// TODO: rework once it is clearer how the process actually works + +The unattended installation can help to automate the installation process from +the very beginning. It needs the dedicated ISO image for unattended +installations. + + +For the automatic installation, the options that the regular installer would +ask for, need to be provided in an answer file. The answer file can be placed +on a USB flash drive. The volume needs to be labeled 'PROXMOXINST' and needs to +contain the answer file named 'answer.toml'. + +The answer file allows to match to select the network card and disks used for +the installation by certain properties of the devices. + +[[installation_auto_answer_file]] +Answer file +~~~ + +The answer file is expected in `TOML` format. The following example shows an +answer file that uses the DHCP provided network settings. It will use a ZFS +Raid 10 with an 'ashift' of '12' and will use all Micron disks it can find. + + +[global] +keyboard = "de" +country = "at" +fqdn = "pve-1.example.com" +mailto = "m...@example.com" +timezone = "Europe/Vienna" +password = "123456" + +[network] +use_dhcp = true + +[disks] +filesystem = "zfs-raid10" +zfs.ashift = 12 +filter.ID_SERIAL = "Micron_*" + + +Global Section +^^ + +This section contains the following keys: + +`keyboard`:: The keyboard layout. The following options are possible: + +de de-ch dk en-gb en-us es fi fr fr-be fr-ca fr-ch hu is it jp lt mk nl no pl pt pt-br se si tr + + +`country`:: The country code in the two letter variant. For example `at`, `us`, +or `fr`. + +`fqdn`:: The fully qualified domain of the host. The domain part will be used +as the search domain. + +`mailto`:: The default email address. Used for notifications. + +`timezone`:: The timezone in `tzdata` format. For example `Europe/Vienna` or +`America/New_York`. + +`password`:: The password for the `root` user. + +`pre_command`:: A list of commands to run prior to the installation. + +`post_command`:: A list of commands run after the installation. +// TODO: explain commands and list of available useful CLI tools in the iso + +`reboot_on_error`:: Set to `true` is the installer should automatically reboot +if erros are encountered. The default behavior is to wait to give the +administrator a chance to investigate why the unattended installation failed. + + +Network Section +^^^ + +`use_dhcp`:: Set to `true` if the IP configuration received by DHCP should be +used. + +`cidr`:: IP address in CIDR notation. For example `192.168.1.10/24`. + +`dns`:: IP address of the DNS server. + +`gateway`:: IP address of the default gateway. + +`filter`:: Filter against `UDEV` properties to select the network card. See +xref:installation_auto_filter[Filters]. + + +Disks Section +^ + +`filesystem`:: The file system used for the installation. The options are: +*`ext4` +*`xfs` +*`zfs-raid0` +*`zfs-raid1` +*`zfs-raid10` +*`zfs-raidz1` +*`zfs-raidz2` +*`zfs-raidz3` +*`btrfs-raid0` +*`btrfs-raid1` +*`btrfs-raid10` + +`disk_selection`:: List of disks to use. Useful if you are sure about the disk +names. For example: + + +disk_selection = ["sda", "sdb"] + + +`filter_match`:: Can be `any` or `all`. Decides if a match of any filter is +enough or if all filters need to match for a disk to be selected. Default is `any`. + +`filter`:: Filter against `UDEV` properties to select disks to install to. See +xref:installation_auto_filter[Filters]. Filters won't be used if +`disk_selection` is configured. + +`zfs`:: ZFS specific properties. See xref:advanced_zfs_options[Advanced ZFS Configuration Options] +for more details. The properties are: +* `ashift` +* `checksum` +* `compress` +* `copies` +* `hdsize` + +`lvm`:: Advanced properties that can be used when `ext4` or `xfs` is used as `filesystem`. +See xref:advanced_lvm_options[Advanced LVM Configuration Options] for more details. The properties are: +* `hdsize` +* `swapsize` +* `maxroot` +* `maxvz` +* `minfree` + +`btrfs`:: BTRFS specific settings. Current
[pve-devel] [PATCH v1 installer 08/18] auto-installer: add answer file definition
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/answer.rs | 147 +++
proxmox-auto-installer/src/lib.rs| 1 +
2 files changed, 148 insertions(+)
create mode 100644 proxmox-auto-installer/src/answer.rs
diff --git a/proxmox-auto-installer/src/answer.rs
b/proxmox-auto-installer/src/answer.rs
new file mode 100644
index 000..0f6c593
--- /dev/null
+++ b/proxmox-auto-installer/src/answer.rs
@@ -0,0 +1,147 @@
+use serde::{Deserialize, Serialize};
+use std::collections::BTreeMap;
+
+#[derive(Clone, Deserialize, Debug)]
+#[serde(rename_all = "lowercase")]
+pub struct Answer {
+pub global: Global,
+pub network: Network,
+pub disks: Disks,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct Global {
+pub country: String,
+pub fqdn: String,
+pub keyboard: String,
+pub mailto: String,
+pub timezone: String,
+pub password: String,
+pub pre_command: Option>,
+pub post_command: Option>,
+pub reboot_on_error: Option,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct Network {
+pub use_dhcp: Option,
+pub cidr: Option,
+pub dns: Option,
+pub gateway: Option,
+// use BTreeMap to have keys sorted
+pub filter: Option>,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct Disks {
+pub filesystem: Option,
+pub disk_selection: Option>,
+pub filter_match: Option,
+// use BTreeMap to have keys sorted
+pub filter: Option>,
+pub zfs: Option,
+pub lvm: Option,
+pub btrfs: Option,
+}
+
+#[derive(Clone, Deserialize, Debug, PartialEq)]
+#[serde(rename_all = "lowercase")]
+pub enum FilterMatch {
+Any,
+All,
+}
+
+#[derive(Clone, Deserialize, Serialize, Debug)]
+#[serde(rename_all = "kebab-case")]
+pub enum Filesystem {
+Ext4,
+Xfs,
+ZfsRaid0,
+ZfsRaid1,
+ZfsRaid10,
+ZfsRaidZ1,
+ZfsRaidZ2,
+ZfsRaidZ3,
+BtrfsRaid0,
+BtrfsRaid1,
+BtrfsRaid10,
+}
+
+#[derive(Clone, Deserialize, Debug)]
+pub struct ZfsOptions {
+pub ashift: Option,
+pub arc_max: Option,
+pub checksum: Option,
+pub compress: Option,
+pub copies: Option,
+pub hdsize: Option,
+}
+
+impl ZfsOptions {
+pub fn new() -> ZfsOptions {
+ZfsOptions {
+ashift: None,
+arc_max: None,
+checksum: None,
+compress: None,
+copies: None,
+hdsize: None,
+}
+}
+}
+
+#[derive(Copy, Clone, Debug, Default, Eq, PartialEq, Deserialize)]
+#[serde(rename_all(deserialize = "lowercase"))]
+pub enum ZfsCompressOption {
+#[default]
+On,
+Off,
+Lzjb,
+Lz4,
+Zle,
+Gzip,
+Zstd,
+}
+
+#[derive(Copy, Clone, Debug, Default, Eq, PartialEq, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+pub enum ZfsChecksumOption {
+#[default]
+On,
+Off,
+Fletcher2,
+Fletcher4,
+Sha256,
+}
+
+#[derive(Clone, Deserialize, Serialize, Debug)]
+pub struct LvmOptions {
+pub hdsize: Option,
+pub swapsize: Option,
+pub maxroot: Option,
+pub maxvz: Option,
+pub minfree: Option,
+}
+
+impl LvmOptions {
+pub fn new() -> LvmOptions {
+LvmOptions {
+hdsize: None,
+swapsize: None,
+maxroot: None,
+maxvz: None,
+minfree: None,
+}
+}
+}
+
+#[derive(Clone, Deserialize, Serialize, Debug)]
+pub struct BtrfsOptions {
+pub hdsize: Option,
+}
+
+impl BtrfsOptions {
+pub fn new() -> BtrfsOptions {
+BtrfsOptions { hdsize: None }
+}
+}
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index e69de29..7813b98 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -0,0 +1 @@
+pub mod answer;
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 01/18] tui: common: move InstallConfig struct to common crate
It describes the data structure expected by the low-level-installer.
We do this so we can use it in more than the TUI installer, for example
the planned auto installer.
Make the members public so we can easily implement a custom From method
for each dependent crate.
Signed-off-by: Aaron Lauterer
---
proxmox-installer-common/src/setup.rs | 86 +++-
proxmox-tui-installer/src/setup.rs| 98 +--
.../src/views/install_progress.rs | 4 +-
3 files changed, 90 insertions(+), 98 deletions(-)
diff --git a/proxmox-installer-common/src/setup.rs
b/proxmox-installer-common/src/setup.rs
index 472e1f2..03beb77 100644
--- a/proxmox-installer-common/src/setup.rs
+++ b/proxmox-installer-common/src/setup.rs
@@ -12,7 +12,10 @@ use std::{
use serde::{de, Deserialize, Deserializer, Serialize, Serializer};
use crate::{
-options::{Disk, ZfsBootdiskOptions, ZfsChecksumOption, ZfsCompressOption},
+options::{
+BtrfsRaidLevel, Disk, FsType, ZfsBootdiskOptions, ZfsChecksumOption,
ZfsCompressOption,
+ZfsRaidLevel,
+},
utils::CidrAddress,
};
@@ -387,3 +390,84 @@ pub fn spawn_low_level_installer(test_mode: bool) ->
io::Result
.stdout(Stdio::piped())
.spawn()
}
+
+/// See Proxmox::Install::Config
+#[derive(Serialize)]
+pub struct InstallConfig {
+pub autoreboot: usize,
+
+#[serde(serialize_with = "serialize_fstype")]
+pub filesys: FsType,
+pub hdsize: f64,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub swapsize: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub maxroot: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub minfree: Option,
+#[serde(skip_serializing_if = "Option::is_none")]
+pub maxvz: Option,
+
+#[serde(skip_serializing_if = "Option::is_none")]
+pub zfs_opts: Option,
+
+#[serde(
+serialize_with = "serialize_disk_opt",
+skip_serializing_if = "Option::is_none"
+)]
+pub target_hd: Option,
+#[serde(skip_serializing_if = "HashMap::is_empty")]
+pub disk_selection: HashMap,
+
+pub country: String,
+pub timezone: String,
+pub keymap: String,
+
+pub password: String,
+pub mailto: String,
+
+pub mngmt_nic: String,
+
+pub hostname: String,
+pub domain: String,
+#[serde(serialize_with = "serialize_as_display")]
+pub cidr: CidrAddress,
+pub gateway: IpAddr,
+pub dns: IpAddr,
+}
+
+fn serialize_disk_opt(value: &Option, serializer: S) -> Result
+where
+S: Serializer,
+{
+if let Some(disk) = value {
+serializer.serialize_str(&disk.path)
+} else {
+serializer.serialize_none()
+}
+}
+
+fn serialize_fstype(value: &FsType, serializer: S) -> Result
+where
+S: Serializer,
+{
+use FsType::*;
+let value = match value {
+// proxinstall::$fssetup
+Ext4 => "ext4",
+Xfs => "xfs",
+// proxinstall::get_zfs_raid_setup()
+Zfs(ZfsRaidLevel::Raid0) => "zfs (RAID0)",
+Zfs(ZfsRaidLevel::Raid1) => "zfs (RAID1)",
+Zfs(ZfsRaidLevel::Raid10) => "zfs (RAID10)",
+Zfs(ZfsRaidLevel::RaidZ) => "zfs (RAIDZ-1)",
+Zfs(ZfsRaidLevel::RaidZ2) => "zfs (RAIDZ-2)",
+Zfs(ZfsRaidLevel::RaidZ3) => "zfs (RAIDZ-3)",
+// proxinstall::get_btrfs_raid_setup()
+Btrfs(BtrfsRaidLevel::Raid0) => "btrfs (RAID0)",
+Btrfs(BtrfsRaidLevel::Raid1) => "btrfs (RAID1)",
+Btrfs(BtrfsRaidLevel::Raid10) => "btrfs (RAID10)",
+};
+
+serializer.collect_str(value)
+}
diff --git a/proxmox-tui-installer/src/setup.rs
b/proxmox-tui-installer/src/setup.rs
index 79421d7..e816c12 100644
--- a/proxmox-tui-installer/src/setup.rs
+++ b/proxmox-tui-installer/src/setup.rs
@@ -1,59 +1,11 @@
-use std::{collections::HashMap, fmt, net::IpAddr};
-
-use serde::{Serialize, Serializer};
+use std::collections::HashMap;
use crate::options::InstallerOptions;
use proxmox_installer_common::{
-options::{AdvancedBootdiskOptions, BtrfsRaidLevel, Disk, FsType,
ZfsRaidLevel},
-setup::InstallZfsOption,
-utils::CidrAddress,
+options::AdvancedBootdiskOptions,
+setup::InstallConfig,
};
-/// See Proxmox::Install::Config
-#[derive(Serialize)]
-pub struct InstallConfig {
-autoreboot: usize,
-
-#[serde(serialize_with = "serialize_fstype")]
-filesys: FsType,
-hdsize: f64,
-#[serde(skip_serializing_if = "Option::is_none")]
-swapsize: Option,
-#[serde(skip_serializing_if = "Option::is_none")]
-maxroot: Option,
-#[serde(skip_serializing_if = "Option::is_none")]
-minfree: Option,
-#[serde(skip_serializing_if = "Option::is_none")]
-maxvz: Option,
-
-#[serde(skip_serializing_if = "Option::is_none")]
-zfs_opts: Option,
-
-#[serde(
-serialize_with = "serialize_disk_opt",
-skip_serializing_if = "Option::is_none"
-)]
-target_hd: Option,
-#[serde(skip_serializing_if = "HashMap::is_em
[pve-devel] [PATCH v1 installer 15/18] auto-installer: use glob crate for pattern matching
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/Cargo.toml | 1 +
proxmox-auto-installer/src/utils.rs | 48 +++--
2 files changed, 19 insertions(+), 30 deletions(-)
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 078a333..158a0a8 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -9,6 +9,7 @@ homepage = "https://www.proxmox.com";
[dependencies]
anyhow = "1.0"
+glob = "0.3"
proxmox-installer-common = { path = "../proxmox-installer-common" }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
diff --git a/proxmox-auto-installer/src/utils.rs
b/proxmox-auto-installer/src/utils.rs
index 6e650c5..5990e93 100644
--- a/proxmox-auto-installer/src/utils.rs
+++ b/proxmox-auto-installer/src/utils.rs
@@ -1,4 +1,5 @@
use anyhow::{anyhow, bail, Context, Result};
+use glob::Pattern;
use log::info;
use std::{
collections::BTreeMap,
@@ -21,30 +22,12 @@ use proxmox_installer_common::{
};
use serde::Deserialize;
-/// Supports the globbing character '*' at the beginning, end or both of the
pattern.
-/// Globbing within the pattern is not supported
-fn find_with_glob(pattern: &str, value: &str) -> bool {
-let globbing_symbol = '*';
-let mut start_glob = false;
-let mut end_glob = false;
-let mut pattern = pattern;
-
-if pattern.starts_with(globbing_symbol) {
-start_glob = true;
-pattern = &pattern[1..];
-}
-
-if pattern.ends_with(globbing_symbol) {
-end_glob = true;
-pattern = &pattern[..pattern.len() - 1]
-}
-
-match (start_glob, end_glob) {
-(true, true) => value.contains(pattern),
-(true, false) => value.ends_with(pattern),
-(false, true) => value.starts_with(pattern),
-_ => value == pattern,
-}
+fn find_with_glob(pattern: &str, value: &str) -> Result {
+ let p = Pattern::new(pattern)?;
+ match p.matches(value) {
+ true => Ok(true),
+ false => Ok(false),
+ }
}
pub fn get_network_settings(
@@ -107,7 +90,7 @@ fn get_single_udev_index(
'outer: for (dev, dev_values) in udev_list {
for (filter_key, filter_value) in &filter {
for (udev_key, udev_value) in dev_values {
-if udev_key == filter_key && find_with_glob(filter_value,
udev_value) {
+if udev_key == filter_key && find_with_glob(filter_value,
udev_value)? {
dev_index = Some(dev.clone());
break 'outer; // take first match
}
@@ -132,7 +115,7 @@ fn get_matched_udev_indexes(
let mut did_match_all = true;
for (filter_key, filter_value) in &filter {
for (udev_key, udev_value) in dev_values {
-if udev_key == filter_key && find_with_glob(filter_value,
udev_value) {
+if udev_key == filter_key && find_with_glob(filter_value,
udev_value)? {
did_match_once = true;
} else if udev_key == filter_key {
did_match_all = false;
@@ -465,9 +448,14 @@ mod tests {
#[test]
fn test_glob_patterns() {
let test_value = "foobar";
-assert_eq!(find_with_glob("*bar", test_value), true);
-assert_eq!(find_with_glob("foo*", test_value), true);
-assert_eq!(find_with_glob("foobar", test_value), true);
-assert_eq!(find_with_glob("oobar", test_value), false);
+assert_eq!(find_with_glob("*bar", test_value).unwrap(), true);
+assert_eq!(find_with_glob("foo*", test_value).unwrap(), true);
+assert_eq!(find_with_glob("foobar", test_value).unwrap(), true);
+assert_eq!(find_with_glob("oobar", test_value).unwrap(), false);
+assert_eq!(find_with_glob("f*bar", test_value).unwrap(), true);
+assert_eq!(find_with_glob("f?bar", test_value).unwrap(), false);
+assert_eq!(find_with_glob("fo?bar", test_value).unwrap(), true);
+assert_eq!(find_with_glob("f[!a]obar", test_value).unwrap(), true);
+assert_eq!(find_with_glob("f[oa]obar", test_value).unwrap(), true);
}
}
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 02/18] common: make InstallZfsOption members public
as they will be used directly by the auto installer
Signed-off-by: Aaron Lauterer
---
proxmox-installer-common/src/setup.rs | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/proxmox-installer-common/src/setup.rs
b/proxmox-installer-common/src/setup.rs
index 03beb77..1bc4cb7 100644
--- a/proxmox-installer-common/src/setup.rs
+++ b/proxmox-installer-common/src/setup.rs
@@ -147,13 +147,13 @@ pub fn installer_setup(in_test_mode: bool) ->
Result<(SetupInfo, LocaleInfo, Run
#[derive(Serialize)]
pub struct InstallZfsOption {
-ashift: usize,
+pub ashift: usize,
#[serde(serialize_with = "serialize_as_display")]
-compress: ZfsCompressOption,
+pub compress: ZfsCompressOption,
#[serde(serialize_with = "serialize_as_display")]
-checksum: ZfsChecksumOption,
-copies: usize,
-arc_max: usize,
+pub checksum: ZfsChecksumOption,
+pub copies: usize,
+pub arc_max: usize,
}
impl From for InstallZfsOption {
--
2.39.2
___
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH v1 installer 17/18] auto-installer: add proxmox-installer-filter helper tool
The proxmox-installer-filter tool is a helper utility to fetch UDEV
properties quickly and show them to the user.
Additionally it allows the user to test out filters for the auto
installer answer file to see which devices would be selected.
Since this tool should be able to run outside of the installer
environment, it does not rely on the device information provided by the
low-level installer. It instead fetches the list of disks and NICs by
itself.
The rules when a device is ignored, should match how the low-level
installer handles it.
Signed-off-by: Aaron Lauterer
---
Makefile | 1 +
proxmox-auto-installer/Cargo.toml | 2 +
proxmox-auto-installer/src/answer.rs | 3 +-
.../src/bin/proxmox-installer-filter.rs | 298 ++
4 files changed, 303 insertions(+), 1 deletion(-)
create mode 100644 proxmox-auto-installer/src/bin/proxmox-installer-filter.rs
diff --git a/Makefile b/Makefile
index b724789..2861f9e 100644
--- a/Makefile
+++ b/Makefile
@@ -20,6 +20,7 @@ PREFIX = /usr
BINDIR = $(PREFIX)/bin
USR_BIN := \
proxmox-tui-installer\
+ proxmox-installer-filter\
proxmox-fetch-answer\
proxmox-auto-installer
diff --git a/proxmox-auto-installer/Cargo.toml
b/proxmox-auto-installer/Cargo.toml
index 158a0a8..570c34c 100644
--- a/proxmox-auto-installer/Cargo.toml
+++ b/proxmox-auto-installer/Cargo.toml
@@ -9,9 +9,11 @@ homepage = "https://www.proxmox.com";
[dependencies]
anyhow = "1.0"
+clap = { version = "4.0", features = ["derive"] }
glob = "0.3"
proxmox-installer-common = { path = "../proxmox-installer-common" }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
toml = "0.5.11"
log = "0.4.20"
+regex = "1.7"
diff --git a/proxmox-auto-installer/src/answer.rs
b/proxmox-auto-installer/src/answer.rs
index 0f6c593..de88d7d 100644
--- a/proxmox-auto-installer/src/answer.rs
+++ b/proxmox-auto-installer/src/answer.rs
@@ -1,3 +1,4 @@
+use clap::ValueEnum;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
@@ -44,7 +45,7 @@ pub struct Disks {
pub btrfs: Option,
}
-#[derive(Clone, Deserialize, Debug, PartialEq)]
+#[derive(Clone, Deserialize, Debug, PartialEq, ValueEnum)]
#[serde(rename_all = "lowercase")]
pub enum FilterMatch {
Any,
diff --git a/proxmox-auto-installer/src/bin/proxmox-installer-filter.rs
b/proxmox-auto-installer/src/bin/proxmox-installer-filter.rs
new file mode 100644
index 000..315f414
--- /dev/null
+++ b/proxmox-auto-installer/src/bin/proxmox-installer-filter.rs
@@ -0,0 +1,298 @@
+use anyhow::{bail, Result};
+use clap::{Args, Parser, Subcommand, ValueEnum};
+use glob::Pattern;
+use regex::Regex;
+use serde::{Deserialize, Serialize};
+use std::{collections::BTreeMap, fs, path::PathBuf, process::Command};
+
+use proxmox_auto_installer::{
+answer::FilterMatch,
+utils::{get_matched_udev_indexes, get_single_udev_index},
+};
+
+/// Tool to get info for devices and test filter matches
+#[derive(Parser, Debug)]
+#[command(author, version, about, long_about = None)]
+struct Cli {
+#[command(subcommand)]
+command: Commands,
+}
+
+#[derive(Subcommand, Debug)]
+enum Commands {
+Info(CommandInfo),
+Match(CommandMatch),
+}
+
+/// Helper utility to show device information and test filters for the Proxmox
auto installer
+#[derive(Args, Debug)]
+struct CommandInfo {
+/// For which device type information should be shown
+#[arg(name="type", short, long, value_enum,
default_value_t=AllDeviceTypes::All)]
+device: AllDeviceTypes,
+}
+
+/// Test which devices the given filter match against
+///
+/// Filters support the following syntax:
+/// ? Match a single character
+/// * Match any number of characters
+/// [a], [0-9] Specifc character or range of characters
+/// [!a] Negate a specific character of range
+///
+/// To avoid globbing characters being interpreted by the shell, use single
quotes.
+/// Multiple filters can be defined.
+///
+/// Examples:
+/// Match disks against the serial number and device name, both must match:
+///
+/// proxmox-installer-filter match --filter-match all disk
'ID_SERIAL_SHORT=**' 'DEVNAME=*nvme*'
+#[derive(Args, Debug)]
+#[command(verbatim_doc_comment)]
+struct CommandMatch {
+/// Device type to match the filter against
+//#[arg(name = "type", long, short, value_enum, required = true)]
+r#type: Devicetype,
+
+/// Filter in the format KEY=VALUE where the key is the UDEV key and VALUE
the filter string.
+//#[arg(long, num_args = 1.., verbatim_doc_comment, required = true)]
+filter: Vec,
+
+/// Defines if any filter or all filters must match.
+#[arg(long, value_enum, default_value_t=FilterMatch::Any)]
+filter_match: FilterMatch,
+}
+
+#[derive(Args, Debug)]
+struct GlobalOpts {
+/// Output format
+#[arg(long, short, value_enum)]
+format: OutputFormat,
+}
+
+#[derive(Clone, Debug, Val
[pve-devel] [PATCH v1 installer 10/18] auto-installer: add utils
contains several utility structs and functions.
For example: a simple pattern matcher that matches wildcards at the
beginning or end of the filter.
It currently uses a dedicated function (parse_answer) to generate the
InstallConfig struct instead of a From implementation. This is because
for now the source data is spread over several other structs in
comparison to one in the TUI installer.
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/src/lib.rs | 1 +
proxmox-auto-installer/src/utils.rs | 473
2 files changed, 474 insertions(+)
create mode 100644 proxmox-auto-installer/src/utils.rs
diff --git a/proxmox-auto-installer/src/lib.rs
b/proxmox-auto-installer/src/lib.rs
index 8cda416..72884c1 100644
--- a/proxmox-auto-installer/src/lib.rs
+++ b/proxmox-auto-installer/src/lib.rs
@@ -1,2 +1,3 @@
pub mod answer;
pub mod udevinfo;
+pub mod utils;
diff --git a/proxmox-auto-installer/src/utils.rs
b/proxmox-auto-installer/src/utils.rs
new file mode 100644
index 000..6e650c5
--- /dev/null
+++ b/proxmox-auto-installer/src/utils.rs
@@ -0,0 +1,473 @@
+use anyhow::{anyhow, bail, Context, Result};
+use log::info;
+use std::{
+collections::BTreeMap,
+net::IpAddr,
+process::{Command, Stdio},
+str::FromStr,
+};
+
+use crate::{
+answer,
+answer::{Answer, FilterMatch},
+udevinfo::UdevInfo,
+};
+use proxmox_installer_common::{
+options::{
+BtrfsRaidLevel, FsType, NetworkOptions, ZfsChecksumOption,
ZfsCompressOption, ZfsRaidLevel,
+},
+setup::{InstallConfig, InstallZfsOption, LocaleInfo, RuntimeInfo,
SetupInfo},
+utils::{CidrAddress, Fqdn},
+};
+use serde::Deserialize;
+
+/// Supports the globbing character '*' at the beginning, end or both of the
pattern.
+/// Globbing within the pattern is not supported
+fn find_with_glob(pattern: &str, value: &str) -> bool {
+let globbing_symbol = '*';
+let mut start_glob = false;
+let mut end_glob = false;
+let mut pattern = pattern;
+
+if pattern.starts_with(globbing_symbol) {
+start_glob = true;
+pattern = &pattern[1..];
+}
+
+if pattern.ends_with(globbing_symbol) {
+end_glob = true;
+pattern = &pattern[..pattern.len() - 1]
+}
+
+match (start_glob, end_glob) {
+(true, true) => value.contains(pattern),
+(true, false) => value.ends_with(pattern),
+(false, true) => value.starts_with(pattern),
+_ => value == pattern,
+}
+}
+
+pub fn get_network_settings(
+answer: &Answer,
+udev_info: &UdevInfo,
+runtime_info: &RuntimeInfo,
+setup_info: &SetupInfo,
+) -> Result {
+let mut network_options = NetworkOptions::defaults_from(setup_info,
&runtime_info.network);
+
+info!("Setting network configuration");
+
+// Always use the FQDN from the answer file
+network_options.fqdn = Fqdn::from(answer.global.fqdn.as_str())
+.map_err(|err| anyhow!("Error parsing FQDN: {err}"))?;
+
+if answer.network.use_dhcp.is_none() || !answer.network.use_dhcp.unwrap() {
+network_options.address = CidrAddress::from_str(
+answer
+.network
+.cidr
+.clone()
+.context("No CIDR defined")?
+.as_str(),
+)
+.map_err(|_| anyhow!("Error parsing CIDR"))?;
+network_options.dns_server = IpAddr::from_str(
+answer
+.network
+.dns
+.clone()
+.context("No DNS server defined")?
+.as_str(),
+)
+.map_err(|_| anyhow!("Error parsing DNS server"))?;
+network_options.gateway = IpAddr::from_str(
+answer
+.network
+.gateway
+.clone()
+.expect("No gateway defined")
+.as_str(),
+)
+.map_err(|_| anyhow!("Error parsing gateway"))?;
+network_options.ifname =
+get_single_udev_index(answer.network.filter.clone().unwrap(),
&udev_info.nics)?
+}
+
+Ok(network_options)
+}
+
+fn get_single_udev_index(
+filter: BTreeMap,
+udev_list: &BTreeMap>,
+) -> Result {
+if filter.is_empty() {
+bail!("no filter defined");
+}
+let mut dev_index: Option = None;
+'outer: for (dev, dev_values) in udev_list {
+for (filter_key, filter_value) in &filter {
+for (udev_key, udev_value) in dev_values {
+if udev_key == filter_key && find_with_glob(filter_value,
udev_value) {
+dev_index = Some(dev.clone());
+break 'outer; // take first match
+}
+}
+}
+}
+if dev_index.is_none() {
+bail!("filter did not match any device");
+}
+
+Ok(dev_index.unwrap())
+}
+
+fn get_matched_udev_indexes(
+filter: BTreeMap,
+udev_list: &BTreeMap>,
+match_all: bool,
+) -> Result> {
+let mut matches = vec![
[pve-devel] [PATCH v1 installer 12/18] auto-installer: add tests for answer file parsing
By matching the resulting json to be passed to the low level installer
against known good ones.
The environment info was gathered from one of our AMD Epyc Rome test
servers to have a realistic starting point.
Signed-off-by: Aaron Lauterer
---
proxmox-auto-installer/tests/parse-answer.rs | 102 ++
.../tests/resources/iso-info.json | 1 +
.../tests/resources/locales.json | 1 +
.../resources/parse_answer/disk_match.json| 29 +
.../resources/parse_answer/disk_match.toml| 14 +++
.../parse_answer/disk_match_all.json | 26 +
.../parse_answer/disk_match_all.toml | 16 +++
.../parse_answer/disk_match_any.json | 33 ++
.../parse_answer/disk_match_any.toml | 16 +++
.../tests/resources/parse_answer/minimal.json | 17 +++
.../tests/resources/parse_answer/minimal.toml | 14 +++
.../resources/parse_answer/nic_matching.json | 17 +++
.../resources/parse_answer/nic_matching.toml | 19
.../tests/resources/parse_answer/readme | 4 +
.../resources/parse_answer/specific_nic.json | 17 +++
.../resources/parse_answer/specific_nic.toml | 19
.../tests/resources/parse_answer/zfs.json | 27 +
.../tests/resources/parse_answer/zfs.toml | 19
.../tests/resources/run-env-info.json | 1 +
.../tests/resources/run-env-udev.json | 1 +
20 files changed, 393 insertions(+)
create mode 100644 proxmox-auto-installer/tests/parse-answer.rs
create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json
create mode 100644 proxmox-auto-installer/tests/resources/locales.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/minimal.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/minimal.toml
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml
create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json
create mode 100644
proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml
create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json
create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml
create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json
create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json
diff --git a/proxmox-auto-installer/tests/parse-answer.rs
b/proxmox-auto-installer/tests/parse-answer.rs
new file mode 100644
index 000..7dabe55
--- /dev/null
+++ b/proxmox-auto-installer/tests/parse-answer.rs
@@ -0,0 +1,102 @@
+use std::path::PathBuf;
+
+use serde_json::Value;
+use std::fs;
+
+use proxmox_auto_installer::answer;
+use proxmox_auto_installer::answer::Answer;
+use proxmox_auto_installer::udevinfo::UdevInfo;
+use proxmox_auto_installer::utils::parse_answer;
+
+use proxmox_installer_common::setup::{
+read_json, RuntimeInfo, LocaleInfo, SetupInfo,
+};
+
+fn get_test_resource_path() -> Result {
+Ok(std::env::current_dir()
+.expect("current dir failed")
+.join("tests/resources"))
+}
+fn get_answer(path: PathBuf) -> Result {
+let answer_raw = std::fs::read_to_string(&path).unwrap();
+let answer: answer::Answer = toml::from_str(&answer_raw)
+.map_err(|err| format!("error parsing answer.toml: {err}"))
+.unwrap();
+
+Ok(answer)
+}
+
+fn setup_test_basic(path: &PathBuf) -> (SetupInfo, LocaleInfo, RuntimeInfo,
UdevInfo) {
+let installer_info: SetupInfo = {
+let mut path = path.clone();
+path.push("iso-info.json");
+
+read_json(&path).map_err(|err| format!("Failed to retrieve setup info:
{err}")).unwrap()
+};
+
+let locale_info = {
+let mut path = path.clone();
+path.push("locales.json");
+
+read_json(&path).map_err(|err| format!("Failed to retrieve locale
info: {err}")).unwrap()
+};
+
+let mut runtime_info: RuntimeInfo = {
+let mut path = path.clone();
+path.push("run-env-info.json");
+
+read_json(&path)
+.map_err(|err| format!("Failed to retrieve runtime environment
info: {err}")).unwrap()
+};
+
+let udev_info: UdevInfo = {
+let mut path = path.clon
