fai-make-nfsroot docs and missing reboot
Hello all. FAI noob here, please be patient :) Reading man page for fai-make-nfsroot it seems 'FULL' install is the default. But after having troubles adding Salt repository (could not verify server certificate), I noticed that ca-certificates (installed only in FULL nfsroot) was not present. After using fai-make-nfsroot -k -c FULL it works. That makes me suspect that the actual default is -s . Could the man page be improved by specifying which is the case? Moreover, at the end of the install, after saying there were no errors, FAI asked to press ENTER to reboot, but IIUC that should me automatic, w/o manual confirmation: docs at https://fai-project.org/fai-guide/#_a_id_faiflags_a_fai_flags says that "If no errors occurred, the client will always reboot automatically." and that seems not to be the case (but it's exactly what I wanted: no error = boot into newly installed SO w/o any interaction, while specifying 'reboot' seems to suggest that it reboots also in case of errors). Tks. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
mke2fs hung
Hello all. Seems the first install on (used) bare metal HW found a problem. Previous tests with a VM worked OK. During the first reinstall on a bare metal server that already contained an installed system, mke2fs hung. I switched to a second console to look at the issue and found that stderr tmpfile for mke2fs contains "Found a dos partition table in /dev/sda2" and stdio tmpfile "prompts" with "Proceed anyway? (n/Y)". Giving a 'y' in the main console lets it proceed, but it shouldn't have stopped. It seems the wipefs -af /dev/sda* before parted is not enough. Maybe a second wipefs is needed between parted and mkfs [*]? IIUC it's quite a corner case (new gpt partition overlapping an old dos extended partition), but probably it's better to handle it. [*] https://unix.stackexchange.com/questions/394984/partition-still-encrypted-with-luks-after-wipefs/394999#394999 HIH. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: fai-make-nfsroot docs and missing reboot
Hi Robert. Then it seems the docs are outdated. Not a big issue, even if the described behaviour would be more useful (automatic reboot if no errors, wait for confirmation in case of errors unless 'reboot' is given). :) BYtE, Diego Il 31/05/2022 12:23, Robert Markula ha scritto: Hi Diego, Moreover, at the end of the install, after saying there were no errors, FAI asked to press ENTER to reboot, but IIUC that should me automatic, w/o manual confirmation: docs at https://fai-project.org/fai-guide/#_a_id_faiflags_a_fai_flags says that "If no errors occurred, the client will always reboot automatically." and that seems not to be the case (but it's exactly what I wanted: no error = boot into newly installed SO w/o any interaction, while specifying 'reboot' seems to suggest that it reboots also in case of errors). In your template (e.g. /srv/tftp/fai/pxelinux.cfg/DEFAULT.tmpl or whereever your tftp root lies) you have to add 'reboot' to your 'FAI_FLAGS'. My DEFAULT.tmpl looks like this: # generated by fai-chboot for host default with IP no IP default fai-generated label fai-generated kernel vmlinuz-4.19.0-17-amd64 append initrd=initrd.img-4.19.0-17-amd64 ip=dhcp root=10.12.0.1:/srv/fai/nfsroot:vers=3 rootovl FAI_FLAGS=verbose,sshd,createvt,reboot FAI_CONFIG_SRC=nfs://10.12.0.1/srv/fai/config FAI_ACTION=install 'sshd' allows me to SSH into the machine during install, with 'createvt' I can switch to a dedicated virtual terminal during install and 'reboot' instructs FAI to reboot at the end of the installation process instead of waiting for someone to press 'enter'. Robert -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: mke2fs hung
Tks Robert.
The issue I had was a bit less extreme :)
md and lvm devices are another can of worms, but luckily I don't use 'em
often :)
BYtE,
Diego
Il 01/06/2022 12:31, Robert Markula ha scritto:
I switched to a second console to look at the issue and found that
stderr tmpfile for mke2fs contains "Found a dos partition table in
/dev/sda2" and stdio tmpfile "prompts" with "Proceed anyway? (n/Y)".
Giving a 'y' in the main console lets it proceed, but it shouldn't
have stopped.
It seems the wipefs -af /dev/sda* before parted is not enough. Maybe a
second wipefs is needed between parted and mkfs [*]?
IIUC it's quite a corner case (new gpt partition overlapping an old
dos extended partition), but probably it's better to handle it.
In the past I stumbled across that issue as well. So I created a hook
'mountdisks.DANGEROUS' that includes, among others, the following lines:
# Clear any MD arrays:
if [ $(grep md0 /proc/mdstat) ]; then
mdadm --stop /dev/md0
fi
if [ $(grep md1 /proc/mdstat) ]; then
mdadm --stop /dev/md1
fi
if [ $(grep md /proc/mdstat) ]; then
# Clear the whole disks:
mdadm --zero-superblock --force $DISK_A
# Clear arrays using a partition (e.g. a swap partition):
mdadm --zero-superblock --force ${DISK_A_SWAP}
fi
# Clear the partition table:
sgdisk --zap-all $DISK_A
Somehow redundant, I know, but I had issues with mdadm before. Never had
a problem ever since. But be careful - this ensures that the disk gets
completely wiped and no partition is preserved, even if you have a
'preserve' statement in your disk_config.
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Transient secrets
Hi all. Is there a preferred way to pass a (different) secret to every host being installed? Something to implement a workflow like: - admin asks Salt to (re)install a host - salt handles shutdown and switch reconfiguration (OT) - salt tells FAIserver to enable install of given host - FAI generates the secret and passes it back to Salt (or Salt generates the secret and passes it to FAI, as long there's a shared secret) - the host boots via network and installs as usual, saving/using the given secret - FAI (or the reinstalled host) tells Salt reinstall is complete and Salt "cleans up" (reconfig switches & so on) (OT) The only "solution" I could find is to save the secret in /srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS, FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 255 chars there's not much space... I's good just for very small "secrets" (that gets transferred in the clear, hence the need to reconfigure the switches). -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Transient secrets
Hi Andrew. That's an option, but is seems less secure: while PXE net have to be quite "locked down", NFS could potentially be exposed on a "public" network (say to handle reinstalls on many networks with a single server). If only machines had an "attestation key" by default... Maybe an USB key to insert in the machine being reinstalled... Possibly in an internal slot... Uhm... Still brainstorming... Tks, Diego Il 07/07/2022 08:22, Andrew Ruthven ha scritto: Hey, I'm not sure if this is preferred or not, but the approach I take is to have a command we run first, that copies any required secrets (and will generate SSH host keys and puppet certs if required first) into the NFS root. A cron job runs every 15 minutes and cleans up any of those secrets which are older than 2 hours (this could be much shorter). Cheers, Andrew On Thu, 2022-07-07 at 08:12 +0200, Diego Zuccato wrote: Hi all. Is there a preferred way to pass a (different) secret to every host being installed? Something to implement a workflow like: - admin asks Salt to (re)install a host - salt handles shutdown and switch reconfiguration (OT) - salt tells FAIserver to enable install of given host - FAI generates the secret and passes it back to Salt (or Salt generates the secret and passes it to FAI, as long there's a shared secret) - the host boots via network and installs as usual, saving/using the given secret - FAI (or the reinstalled host) tells Salt reinstall is complete and Salt "cleans up" (reconfig switches & so on) (OT) The only "solution" I could find is to save the secret in /srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS, FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 255 chars there's not much space... I's good just for very small "secrets" (that gets transferred in the clear, hence the need to reconfigure the switches). -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Transient secrets
The more I think about it, the more I convince myself that an USB key (preferably connected to the internal USB connector) could be quite a good compromise: cannot be stolen too easily (requires opening the chassis), can be installed w/o requiring special skills, is cheap, and stores "more than enough" :) Now I only have to figure out how to reliably detect its presence during install, then it's just matter of copying files. Even better could be a "virtual key" connected via IPMI storage when needed (even if data gets transferred in cleartext, it's over a "secured network": only a fool would expose IPMI interface to unsecure networks!)... Il 07/07/2022 08:55, Andrew Ruthven ha scritto: Hey, Yes, agreed, depends on the use case. For the gear I'm dealing with they're on physically very secure networks and NFS is firewalled off. You could potentially have a kernel token as you suggest and then go to fetch the secrets from a HashiCorp Vault with an approval needing to be issued. I know of someone who used to store GPG encrypted tar files in their FAI profile and you had to enter a GPG key during the build to decrypt them. We do in some cases generate passwords (root and encrypted filesystems) during build and have those emailled with GPG encryption to the relevant parties. Cheers, Andrew On Thu, 2022-07-07 at 08:35 +0200, Diego Zuccato wrote: Hi Andrew. That's an option, but is seems less secure: while PXE net have to be quite "locked down", NFS could potentially be exposed on a "public" network (say to handle reinstalls on many networks with a single server). If only machines had an "attestation key" by default... Maybe an USB key to insert in the machine being reinstalled... Possibly in an internal slot... Uhm... Still brainstorming... Tks, Diego Il 07/07/2022 08:22, Andrew Ruthven ha scritto: Hey, I'm not sure if this is preferred or not, but the approach I take is to have a command we run first, that copies any required secrets (and will generate SSH host keys and puppet certs if required first) into the NFS root. A cron job runs every 15 minutes and cleans up any of those secrets which are older than 2 hours (this could be much shorter). Cheers, Andrew On Thu, 2022-07-07 at 08:12 +0200, Diego Zuccato wrote: Hi all. Is there a preferred way to pass a (different) secret to every host being installed? Something to implement a workflow like: - admin asks Salt to (re)install a host - salt handles shutdown and switch reconfiguration (OT) - salt tells FAIserver to enable install of given host - FAI generates the secret and passes it back to Salt (or Salt generates the secret and passes it to FAI, as long there's a shared secret) - the host boots via network and installs as usual, saving/using the given secret - FAI (or the reinstalled host) tells Salt reinstall is complete and Salt "cleans up" (reconfig switches & so on) (OT) The only "solution" I could find is to save the secret in /srv/tftp/fai/pxelinux.cfg/C0A8xxyy in append line, like FAI_FLAGS, FAI_CONFIG_SRC and FAI_ACTION, but since append line can be at most 255 chars there's not much space... I's good just for very small "secrets" (that gets transferred in the clear, hence the need to reconfigure the switches). -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz <mailto:and...@etc.gen.nz> | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz <https://catalystcloud.nz> | -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Secure deploy of keys
Hello all. What's the recommended way to deploy (or re-deploy) security-sensitive objects (just to say one: private ssh key to avoid client warnings when redeploying a server)? TIA -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Tks. Too bad I fear it's not applicable to my scenario. First because the network is public. Second because ssh is just one of the secrets I have to distribute (others are usually SaltStack key and Gluster certificate). I'm thinking that probably this is one of the few cases where a TPM is actually useful... GPG encrypted tarballs can be a good solution if there's a trusted person that can insert the password (or a tpm that can decrypt it) to complete the install... Diego Il 13/12/2022 20:44, Andrew Ruthven ha scritto: Hey, On Tue, 2022-12-13 at 14:47 +0100, Diego Zuccato wrote: What's the recommended way to deploy (or re-deploy) security-sensitive objects (just to say one: private ssh key to avoid client warnings when redeploying a server)? For things like ssh host keys I have a command that we run which copies them into the NFSROOT, and then a cron job that runs every minute that removes "expired" files from the NFSROOT. Given our NFSROOT is on a restricted network I feel that is sufficient. I know someone who had GPG encrypted tarballs, but that required entering a passphrase during the build process. Another option for ssh which I am considering is using PKI for it. Then servers and clients just need to trust a CA. Cheers, Andrew -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Il 15/12/2022 18:15, Toomas Tamm via linux-fai ha scritto: Some things that I can imagine that could mitigate such risks would be: - Inputting some secret on the physical machine during install (from the keyboard, USB stick, etc). This would defeat the idea of "fully automatic" install. That's a form of "root of trust". - Pre-loading a secret onto hardware (is this what you mean by using TPM?). Yes. TPM (Trusted Platform Module) is a piece of HW that handles crypto keys and should be hard to tamper. At least it would require unsupervised physical access to the interior of the machine for quite a long time. But once the attacker does have unsupervised physical access to the machine, it would be faster to just boot from USB key and extract the files. Unless TPM is also used for secure boot, but that's another can of worms. - Time-limiting the availability of secrets and/or some component of FAI. Most of us probably do not install clients every day, all day. That shouldn't be too hard. Just make secrets available only during install. Once the machine is installed it calls a hook to close the access to the secrets. - Monitoring of installation processes and flagging abnormal activities. This would not prevent successful attacks, but possible breaches could be patched up, eg keys replaced afterwards. This seems harder. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Tks for the answer. Sorry for seeing it late but it went in the spam folder :( I didn't know clevis/tang, but it's really interesting (maybe a bit overkill in my scenario). Diego Il 15/12/2022 18:53, Robert Markula ha scritto: Am 15.12.22 um 18:15 schrieb Toomas Tamm via linux-fai: This message was wrapped to be DMARC compliant. The actual message text is therefore in an attachment. Hi Toom, unforunately I can't quote you directly, but regarding a rogue attacker mimicking the MAC of an install client: You have to manually enable a FAI installation, otherwise the client cannot be installed: fai-chboot -c DEFAULT client.example.com Granted, with the right timing one could be faster with a rogue client than with the real client. But on the other hand, any client with access to the FAI NFS server can manually mount the NFSroot and obtain any secrets living on the NFS server via this method. So keeping a secret on the NFSroot is not a viable solution. But there are possibilities to work around that. What has been discussed: 1. the secret is created on the install client during installation and transfered to another system in a secure way, e.g. via SSH 2. the secret is pulled from a third-party solution, which is outside the scope of FAI (e.g. via Salt, Cfengine or any other configuration management software). Authenticated registration of the install client to the configuration management software of your choice is the weakest link here [1] 3. using public key encryption (GPG, PKI, SSH) [2] 4. using a zero-trust-like approach to secrets like clevis/tang [3] I have not looked into solutions like HashiCorp Vault, but maybe that can be cleverly integrated as well? Kind regards, Robert [1] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg07955.html [2] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08003.html [3] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08005.html -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Just did a quick test. Seems feasible to use clevis w/ tpm2 to securely bind credentials to a machine. The idea is: - in case of new install there are no machine-specific files - secrets gets generated as usual - once the machine is up & running, use ssh to run a script to encrypt the needed secret files using machine's TPM and tranfer encrypted files to FAI - in case of reinstall, FAI transfers encrypted files to the machine and runs clevis decrypt to restore 'em That's just a rough idea. Any evident issues? Diego Il 16/01/2023 14:12, Diego Zuccato ha scritto: Tks for the answer. Sorry for seeing it late but it went in the spam folder :( I didn't know clevis/tang, but it's really interesting (maybe a bit overkill in my scenario). Diego Il 15/12/2022 18:53, Robert Markula ha scritto: Am 15.12.22 um 18:15 schrieb Toomas Tamm via linux-fai: This message was wrapped to be DMARC compliant. The actual message text is therefore in an attachment. Hi Toom, unforunately I can't quote you directly, but regarding a rogue attacker mimicking the MAC of an install client: You have to manually enable a FAI installation, otherwise the client cannot be installed: fai-chboot -c DEFAULT client.example.com Granted, with the right timing one could be faster with a rogue client than with the real client. But on the other hand, any client with access to the FAI NFS server can manually mount the NFSroot and obtain any secrets living on the NFS server via this method. So keeping a secret on the NFSroot is not a viable solution. But there are possibilities to work around that. What has been discussed: 1. the secret is created on the install client during installation and transfered to another system in a secure way, e.g. via SSH 2. the secret is pulled from a third-party solution, which is outside the scope of FAI (e.g. via Salt, Cfengine or any other configuration management software). Authenticated registration of the install client to the configuration management software of your choice is the weakest link here [1] 3. using public key encryption (GPG, PKI, SSH) [2] 4. using a zero-trust-like approach to secrets like clevis/tang [3] I have not looked into solutions like HashiCorp Vault, but maybe that can be cleverly integrated as well? Kind regards, Robert [1] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg07955.html [2] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08003.html [3] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08005.html -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Automatically disabling PXE boot
Hello all. I'm returning to FAI after being forced to use xCat (long & sad story... short version: I didn't like it). One of the features of xCat (possibly the only one I liked...) is that I can avoid touching BIOS configuration on the client, leaving it to always try PXE first and fallback to HDD when PXE fails. That does have the advantage that I don't need IPMI or a tech near a crashed system: it's enough to have someone that can press the 'reset' button (or a cheap switched plug). IIUC hooks are run on the system being installed, so I could use LAST hook to somehow signal FAI host to run "fai-chboot -d host". But that would leave DHCP server sending a DHCP OFFER for a PXE boot that's been disabled. Maybe I'm reinventing the wheel, but couldn't find anything. Any hints? TIA. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Automatically disabling PXE boot
Hi Andrew. That would be OK, but I don't need (and it's actually undesirable) to reinstall at every reboot: one of the systems actually requires an extra reboot to complete the setup... if installation restarts, I'll find the system in a bootloop... Or maybe I didn't understand and you're calling fai-chboot and just not bothering about DHCP ? Diego Il 07/06/2023 09:57, Andrew Ruthven ha scritto: Hey, On Wed, 2023-06-07 at 09:45 +0200, Diego Zuccato wrote: IIUC hooks are run on the system being installed, so I could use LAST hook to somehow signal FAI host to run "fai-chboot -d host". But that would leave DHCP server sending a DHCP OFFER for a PXE boot that's been disabled. Maybe I'm reinventing the wheel, but couldn't find anything. We just leave our servers to PXE boot. It slows boot down a little, but we don't reboot that often, so, meh. The reduction in mucking around required to enable PXE boot is worth a slightly slow boot in my opinion. Cheers, Andrew -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Automatically disabling PXE boot
Tks. Quite clear & useful. Diego Il 07/06/2023 12:57, Andrew Ruthven ha scritto: On Wed, 2023-06-07 at 10:05 +0200, Diego Zuccato wrote: Hi Andrew. That would be OK, but I don't need (and it's actually undesirable) to reinstall at every reboot: one of the systems actually requires an extra reboot to complete the setup... if installation restarts, I'll find the system in a bootloop... Or maybe I didn't understand and you're calling fai-chboot and just not bothering about DHCP ? Oh, we're not doing a reinstall on each boot, that'd suck. While we aren't using fai-chboot, similar concept. We have a pxelinux.cfg/default file that has: default localboot label localboot localboot 0 So by default the boxes will just boot off the local disk. -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
os-prober warning considered error?
Hi. I just noticed that FAI installs were waiting at the end because of a single line in error.log: shell.log:Warning: os-prober will not be executed to detect other bootable partitions. I've added it to globalignorepatterns in hooks/savelog.LAST.sh (from 'Warning' to the end). Now I get "Congratulations! No errors found in log files" but task_faiend still prompts for Enter key to reboot. What did I miss? Specifying "reboot" flag seems wrong, since it forces reboot even in case of errors, IIUC. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: os-prober warning considered error?
Seems I still missed the little patch that have to be applied to savelog.LAST.sh hook (adding "export flag_reboot=1" after printing the congrats message). Diego Il 08/06/2023 15:22, Diego Zuccato ha scritto: Hi. I just noticed that FAI installs were waiting at the end because of a single line in error.log: shell.log:Warning: os-prober will not be executed to detect other bootable partitions. I've added it to globalignorepatterns in hooks/savelog.LAST.sh (from 'Warning' to the end). Now I get "Congratulations! No errors found in log files" but task_faiend still prompts for Enter key to reboot. What did I miss? Specifying "reboot" flag seems wrong, since it forces reboot even in case of errors, IIUC. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Installation of package_config/CLASS.gpg
I placed 'em under
/srv/salt/_files/etc/apt/keyrings/-archive-keyring.gpg and
repositories have
deb [signed-by=/etc/apt/keyrings/-archive-keyring.gpg arch=amd64]
https://...
gluster.sls uses:
-8<--
create-keyrings-dir:
file.directory:
- name: /etc/apt/keyrings/
- user: root
- group: root
- mode: 755
add-gluster-key:
file.managed:
- name: /etc/apt/keyrings/gluster-archive-keyring.gpg
- source: salt://_files/etc/apt/keyrings/gluster{{
salt['pillar.get']('gluster_version','') }}-archive-keyring.gpg
add-gluster-repo:
file.managed:
- name: /etc/apt/sources.list.d/gluster.list
- source: salt://_files/etc/apt/sources.list.d/gluster{{
salt['pillar.get']('gluster_version','') }}-{{ grains['oscodename'] }}.list
-8<--
(actually create-keydirs-dir is in a separate sls that gets included by
all sls files that need to add keyrings, but it's just a detail).
Diego
Il 22/08/2023 09:46, Thomas Lange ha scritto:
I would suggest you are using a hook with an fcopy command to put
those files to some other locations.
On Tue, 18 Jul 2023 21:36:04 +1200, Andrew Ruthven said:
> Hey,
> I see that FAI since 5.8.7 will install package_config/CLASS.gpg
> into /etc/apt/trusted.gpg.d/ . Apt will then trust all the keyrings in
> /etc/apt/trusted.gpg.d . This isn't really ideal, and I'd prefer to use
> Signed-By to specify which GPG keyring to trust for our various
additional
> repositories.
> How about having task_repository check for another file, say
> package_config/CLASS.gpg_dest that'd allow us to specify where to copy
> package_config/CLASS.gpg to?
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: RAID + UEFI
TL;DR: IMO having /boot/uefi on (sw) RAID1 is not worth the headaches. Long version follows. That's quite normal. You should use older RAID format that places metadata at the end of the partition instead of the newer that places it at the beginning. And even so it's risky, since one of the partitions could get modified by other systems (the BIOS, for example) that aren't RAID-aware (that's also one of the reasons that lead to the newer format...). Since you're using FAI, it's probably faster to just completely reinstall the system (possibly preserving data partition(s)) than risking RAID desync and related headaches. To keep the disk partitions aligned, you could use the "mirror" space for an extra swap partition :) Diego Il 20/09/2023 09:26, Andrew Ruthven ha scritto: Hey, Just a note that I've found that some hardware doesn't like /boot/efi being on RAID1. Which is rather disappointing. But otherwise, the examples that Justin provided look very similar to what I've succesfully used (although sometimes without the RAID1 for /boot/efi!) Cheers, Andrew -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
FAI + SaltStack anybody?
Hello all. Does someone use FAI to install the base system that will be managed by Salt? I'm trying to integrate 'em but there's still something that doesn't "click"... My current idea is to use Salt to orchestrate the install, but maybe it's better left to FAI? How can I "pass around" minion key so I don't have to manually re-approve the new key every time? The ideal scenario would be: target generates its keypair, sends the pubkey to FAI that "certifies" it's from the system being installed and passes it to Salt. Should I write a custom fai-monitor (that would be needed anyway to disable netboot once system is reinstalled)? TIA. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
Il 05/10/2023 15:17, Carsten Aulbert ha scritto: we usually try with the hardware level configuration being the "border", i.e. everything related to partitioning, initial OS install, at least initial networking set-up is done with FAI (well, and salt is installed configured as well). Ok, that's good. Then FAI reboots the server and upon service start, the server starts a highstate and performs the remaining configuration. Ok, no problem here. To set-up salt, we wrote our own script around fai-chboot which ssh into the salt-master, creates a keypair and copies the files to the appropriate places. Uhm... I don't really like that ssh step. But probably can be straightened out making salt get the pubkey from FAI's state. FAI will install the private key during the installation and the public key is already known on the master, no need to accept the keys anymore. I like even less that the private key is passed from FAI to the target, I'd prefer to only pass back the pubkey. Does that help a bit? Yes, tks. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
Il 05/10/2023 15:54, Laura Smith via linux-fai ha scritto: Its been a while since I worked with Salt, but IIRC it sounds like what is not "clicking" is that you need to fix the TOFU problem. Actually there are 2 distinct problems: - pass the pubkey from the minion to FAI during the install (possibly in an authenticated way) - authorize that key in Salt from FAI Looking back through my notes, it seemshttps://docs.saltproject.io/en/latest/topics/tutorials/multimaster_pki.html might be worth a read. I don't understand. In my scenario, FAI is not a Salt master. And I don't see how making it one could help. It would only double the burden. In particular, maybe "master_sign_pubkey: True" on the Salt master, "verify_master_pubkey_sign: True" on the minion, and the master pubkeys put in "/etc/salt/pki/minion/" on the minions. Then on Salt master all you have to do is approve the new connections as they come online. I'd have to approve on *both* masters. :( -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
Il 05/10/2023 16:58, Sinh Lam ha scritto: You can essentially establish a ’trust’ to auto-accept keys. Then you wouldn’t really have to worry about moving the minion keys around. Once your bootstrap/installation is done, have it run a state to remove the key or auto-purge it somehow. Uh? If the minion is not known to the master, it doesn't receive pillars. And can't interact with the master. Chicken and egg. Honestly I would just leave the base install and anything else that needs to be set up to FAI and run salt against the booted up server after FAI is done and the server has been rebooted. That's what I was planning to do. But without extra "glue" I'm losing context. In particular if FAI tells Salt "I'm having *this* machine reinstalled and its key is this" then Salt can auto-accept that key. But if the machine is not being reinstalled by FAI, there's no reason to auto accept a new key: it could be anybody! Does FAI use protected connections (given that usually there's no available "root of trust" stronger than the MAC address...) to the machine being installed? -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
Il 06/10/2023 10:36, Sinh Lam ha scritto: Reading through your original post - I think there might be some confusion as to what SaltStack does and what FAI does (if not, I apologize). SaltStack is a configuration management tool that is normally used to ensure the target minion's configuration is exactly as it should, while FAI is a provisioning tool that essentially builds the server that SaltStack is used to manage. No need to apologize even if I already knew the difference between FAI and Salt :) With the above said, I do not see what you mean there is a chicken and the egg problem. To approve a minion key, Salt does have to trust the request is coming from the right minion, but it can't know till the key is approved. I do not believe that salt can do the actual installation of the server’s OS because there’s no minion running to begin with. You should really leave that to FAI. Yes, sure. Other tools (like xCat) try to do both and are IMVHO way less versatile. Your concern was how to move the minion around servers that are getting provisioned/re-provisioned so you don’t have to approve the minion each time and I’m sure there’s a couple of ways to do this but right now I see two : 1) turn on auto-accept - you don’t have to worry about approving any minions because they’ll be auto-approved Can't do that on public networks. [*] 2) issue a call to the salt master to accept the new minion when it is registered during fai. This involves you knowing the minion id/name of the key. That's what I'd like to make FAI do. If only there was a 'hook' system on FAI server, triggering scripts at the different stages... The nearest thing I could think of is a custom fai-monitor but it seems quite unsafe :( This is how I’m planning on using SaltStack with FAI - I have a dedicated network that is tightly controlled so to that point I know what connects to it and I know why those servers are connected to that particular network. In essence, I trust this particular network. Because of this, I have auto-accept turned on my salt master. Can't do that: my networks are often exposed. The alternative would mean having to dynamically reconfigure switches to move ports to different VLANs... Quite a big can of worms on its own :( I have FAI install the base os on the server, toward the end of the process, a couple things will happen: * a script will be used to auto-register this server with our CMDB * a script will be used to enroll this minion with the salt-master and set the minion_id (if needed). How does your script authenticate the requests? Or are you just relying on the "secured network" to bypass authentication? [...] That’s it. FAI does the OS (and handles the registering of the server with our CMDB and the minion with the master), and salt stack takes care of the configuration of the server. That's the desiderable outcome :) The glue I believe you’re talking about is a source of truth to populate pillar data and grains so your salt states can actually do something useful. No, that's already taken care of :) And MAC addresses can be spoofed quite easily, so you really shouldn’t rely on that as your ‘root of trust’. I deal with a lot of VMs and each one of those VMs I can easily specify whatever MAC address I want (you really shouldn’t). But spoofing a MAC while it’s in the early parts of pxe/net boot process is harder (if not impossible), you still shouldn’t use it as the ‘root of trust’. Yup, I know. Already did it in DOS :) But stronger authentication either requires TPM or interaction. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
I really like it a lot! Not bulletproof but more secure than a file. Still no way to have "hooks" run on FAI server? Diego Il 06/10/2023 11:18, Thomas Lange ha scritto: On Fri, 06 Oct 2023 21:57:28 +1300, Andrew Ruthven said: > This isn't ideal as the secrets are still present in the NFSROOT for a short > period of time, but does solve the chicken and egg issue others mentioned This reminds me of a solution I once saw. Put some info into a fifo (named pipe), so only one receiver can read it. After that the fifo is empty. What about having a daemon on the FAI server which serves some secrect using: echo secrect | nc -p 12345 -l So only one FAI client can read the secrect from port 12345 once. This may help a little bit. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
Il 06/10/2023 15:15, Johan Beisser ha scritto: With that, on the salt-master, either autoaccept, or find a way to place the minion's public key in `/etc/salt/pki/master/minions/` and that will bypass the key acceptance entirely. Keys, inside of salt, are just managing where the file sits under the various minion directories in `/etc/salt/pki/master/` after all. Yup. that's exactly where my problem lies: that "find a way" is what I'm looking for :) Don't have to do it if you set the master's public key, and minion keys, before the minion is started though. Well, for the minion it's not a problem, as long as it finds the correct pubkey: if its key is missing, a keypair can be generated. But the master doesn't know this new key (yet). Then it's just having a single job starting after FAI's reboot, and doing `salt-call state.highstate` on first boot. It's not a Salt problem, it's just a "timing issue" that I have to understand well. Once Salt knows a minion is being reinstalled (ideally I triggered it applying a given state), it should sync with FAI to wait the moment the minion is rebooting (or, even better, it receives the minion key before the reboot) and knows it can trust that key. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: FAI + SaltStack anybody?
Yes. They're different tools with different objectives. FAI excels at reinstalling a system, but is not a configuration manager: say you have a webserver (actually 3: dev, test and prod) and you need to change the PHP version in use. Sure, you can reinstall from scratch with FAI, but why? Way faster to just update packages. Or are you reinstalling every time there are system updates instead of using apt full-upgrade? Diego Da: linux-fai per conto di Henning Glawe Inviato: venerdì 6 ottobre 2023 17:21 A: fully automatic installation for Linux Oggetto: Re: FAI + SaltStack anybody? Moin, On Thu, Oct 05, 2023 at 02:59:40PM +0200, Diego Zuccato wrote: > Does someone use FAI to install the base system that will be managed by > Salt? Do you have a concrete reason for introducing Salt on top of FAI? FAI can be used to do most of your configuration management via ``fai softupdate`` -- Mit freundlichen Grüßen Henning Glawe Dr. Henning Glawe Max-Planck-Institut für Struktur und Dynamik der Materie Geb. 99 (CFEL), Luruper Chaussee 149, 22761 Hamburg, Germany http://www.mpsd.mpg.de/, Email: henning.gl...@mpsd.mpg.de Building/Room: 99/O2.100, Phone: +49-40-8998-88392
Re: FAI + SaltStack anybody?
Il 06/10/2023 18:33, Matthew Pounsett ha scritto: You could store the public keys that FAI generates in a repository on the FAI server, and have it trigger a Salt webhook to tell the master when it needs to retrieve and install new ones. I'll have to have a look at webhooks. Didn't considere 'em. Could trigger a script that uses salt-cloud to provision the node... Too many ideas :) -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Single FAI server, multiple Debian versions?
Hello all. Is it possible to use a single FAI server to install multiple Debian releases (to different machines, obv)? I'm currently installing bullseye, but I'd like to start testing bookworm deployments. I can't find a howto for setting up multiple NFSROOTs (better if with no changes to the current one, to avoid breaking the working setup). -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Single FAI server, multiple Debian versions?
Tks for the fast answer. I'll have to dig a bit deeper (never used debootstrap explicitly), so it will take a bit more to fully understand. Diego Il 16/01/2024 10:43, Henning Glawe ha scritto: Moin, On Tue, Jan 16, 2024 at 10:22:42AM +0100, Diego Zuccato wrote: Is it possible to use a single FAI server to install multiple Debian releases (to different machines, obv)? I'm currently installing bullseye, but I'd like to start testing bookworm deployments. I can't find a howto for setting up multiple NFSROOTs (better if with no changes to the current one, to avoid breaking the working setup). You can install multiple debian (and even ubuntu) releases from the same nfsroot, if you run ``debootstrap`` at installtime. All you have to do is to delete the pre-built base tarball after building the nfsroot and provide the right settings in your fai config. We use a hook --- /etc/fai/nfsroot-hooks/50remove-base-tar --- #!/bin/bash rm -f $NFSROOT/var/tmp/base.t* and set, depending on the target host classes ``$FAI_DEBOOTSTRAP`` and ``$FAI_DEBOOTSTRAP_OPTS``, e.g.: --- $FAI/class/BOOKWORM.var #!/bin/bash FAI_DEBOOTSTRAP="bookworm http://mpsd-deb-bookworm.desy.de/debian"; FAI_DEBOOTSTRAP_OPTS="--include=aptitude,gnupg" FAI then runs debootstrap with the given options when installing. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Single FAI server, multiple Debian versions?
Tks, that's indeed way easier. And more manageable, especially if the files are kept on a dedicated http server => no changes to nfsroot. The con is that the file needs to be saved somewhere on the local system before being extracted, and that could be a problem with small disks, while with nfsroot it is accessible as a normal file and does not need to be copied. Will dig this, too. For now I'm trying to get rid of some spurious error messages (related to PCI addresses that are not available, probably a BIOS issue) that prevent install to automatically reboot. Diego Il 16/01/2024 10:59, Andrew Ruthven ha scritto: You can install multiple debian (and even ubuntu) releases from the same nfsroot, if you run ``debootstrap`` at installtime. It is much faster if you use basefiles, and have one per release you install. See https://fai-project.org/fai-guide/ and search for basefiles. We set a class of $RELEASE_$ARCH and use that to select the basefile. Cheers, Andrew -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Single FAI server, multiple Debian versions?
Il 16/01/2024 15:00, Thomas Lange ha scritto: > Tks, that's indeed way easier. And more manageable, especially if the > files are kept on a dedicated http server => no changes to nfsroot. > The con is that the file needs to be saved somewhere on the local system > before being extracted, and that could be a problem with small disks, No. The tar file is downloaded into a directory that is a RAM disk. Good. I noticed after answering that basefiles are well under 100MB. But now the install is saying that it's downloading bullseye packages even if I specified class BOOKWORM64. Surely I've messed up something. Work for tomorrow :) Tks for all the help! -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Single FAI server, multiple Debian versions?
Il 16/01/2024 16:20, Robert Markula ha scritto:
Am 16.01.24 um 16:13 schrieb Diego Zuccato:
But now the install is saying that it's downloading bullseye packages
even if I specified class BOOKWORM64. Surely I've messed up something.
Work for tomorrow :)
Have a look at your class/DEBIAN file in your FAI config space. There
should be a line that says:
release=bookworm
I have a DEBIAN.var file that also defines some other variables.
Alternatively, if you want to be able to install multiple distributions
or versions, you can create your own class/BOOKWORM64.var file. It just
needs to contain the 'release'-line from above.
I copied DEBIAN.var to BOOKWORM64.var, then changed the var to
release=bookworm .
I also defined APTPROXY line to be sure file got read. And Actually it
is. But in the console I get a
fcopy: no matching file for any class for etc/apt/sources.list defined
so it seems it copies the one from nfsroot (it should be handled by
scripts/LAST/50-misc , but isn't LAST a bit late?).
So I also added hooks/task_repository/repository.BOOKWORM64 containing:
-8<--
#!/bin/bash
echo < $target/etc/apt/sources.list
# Created by $0
deb $apt_cdn/debian $release main contrib non-free
deb $security_cdn/debian-security ${secsuite} main contrib non-free
EOF
-8<--
But it seems it's not the correct solution, since task_updatebase still
fetches from bullseye :(
Hints?
Tks.
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Single FAI server, multiple Debian versions?
Il 17/01/2024 10:55, Andrew Ruthven ha scritto: On Wed, 2024-01-17 at 09:06 +0100, Diego Zuccato wrote: I copied DEBIAN.var to BOOKWORM64.var, then changed the var to release=bookworm . It'll depend on what you're using as in our profile as well. You need to have a class set that matches the class named used for the basefile you want to use. Are you setting BOOKWORM64? Yup, sure! And in the list printed during boot there are both, with DEBIAN preceding BOOKWORM. I see both var files being read & printed. Since the first workaround didn't work (even after changing 'echo' to 'cat'), I resorted to adding files/etc/apt/sources.list/BOOKWORM64 with the needed lines. At least now the packages get fetched from the right repository, but "it smells worse than my yesterday diaper" (Baby Herman)... -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Accessing external https repo during install
Hello all. I'm trying to pre-install salt from https://repo.saltproject.io . I: - assigned "SALT" class to the host - created hooks/repository.SALT that copies the salt.list, salt-archive-keyring.gpg and /etc/salt/minion.d/master.conf to $target - created package_config/SALT that includes both ca-certificates and salt-minion The problem is that when 'apt update' runs, it doesn't recognize the certificate for the external repo => error gets logged and the salt-minion provided by the distro is installed instead of the one from that external repo. Obvious, since ca-certificates have not yet been installed. How can I have ca-certificates installed when the repository gets added? -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Accessing external https repo during install
Il 17/01/2024 14:15, Carsten Aulbert ha scritto: How can I have ca-certificates installed when the repository gets added? I think you could either add it into your basefile Thought that, but would require regular maintenance, regenerating basefile every time ca-certificates is updated. or add it to your hook to install ca-certificates from Debian first. That whould be the perfect solution. Does that make sense? Sure it does. I just have to understand how to do it the correct way :) First issue (that deranged me): I forgot to set SALT class for the test-fai host, but files/etc/apt/sources.list.d/salt.list/BOOKWORM got copied anyway... some script is fcopy-ing more than expected... Fixed (partially) the first issue, hooks/repository.SALT (the one that should create salt.list file...) finally got called and attempted to install ca-certificate. But it failed. Seems I'm attempting to install it too soon. Uff. Work for tomorrow... Tks for all the hints! -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Accessing external https repo during install
IIUC that's the same as adding 'em to the basefile. Every time an
install errors out, basefile/nfsroot must be regenerated to include
updated root certs. Error prone and time consuming.
I'm now trying to understand:
1) who is copying the whole /etc/apt/sources.list.d during
task_repository, to disable salt.list
2) initialize salt repo with a script later in the configuration phase,
when packages (including ca-certificates) are already installed
Point 1 is really unexpected and shouldn't happen by default. Currently
ruling out it gets done by one of my scripts. Just to be sure:
fcopy /etc/apt/sources
does *not* touch /etc/apt/sources.list.d/, right?
Diego
Il 17/01/2024 17:10, Markus Köberl ha scritto:
On Wednesday, 17 January 2024 16:13:02 CET Diego Zuccato wrote:
Il 17/01/2024 14:15, Carsten Aulbert ha scritto:
How can I have ca-certificates installed when the repository gets added?
I think you could either add it into your basefile
Thought that, but would require regular maintenance, regenerating
basefile every time ca-certificates is updated.
or add it to your
hook to install ca-certificates from Debian first.
That whould be the perfect solution.
Does that make sense?
Sure it does. I just have to understand how to do it the correct way :)
First issue (that deranged me): I forgot to set SALT class for the
test-fai host, but files/etc/apt/sources.list.d/salt.list/BOOKWORM got
copied anyway... some script is fcopy-ing more than expected...
Fixed (partially) the first issue, hooks/repository.SALT (the one that
should create salt.list file...) finally got called and attempted to
install ca-certificate. But it failed. Seems I'm attempting to install
it too soon.
Uff. Work for tomorrow...
Tks for all the hints!
I have on the fai server in /etc/fai/nfsroot.conf:
FAI_DEBOOTSTRAP_OPTS="--include=ca-certificates,apt-transport-https"
and /etc/fai/nfsroot-hooks/ca-certificates:
# load deffinition of ${NFSROOT}
. /etc/fai/nfsroot.conf
mkdir -p ${NFSROOT}/usr/local/share/ca-certificates
cp /etc/fai/nfsroot-hooks/ComodoIntermediateCertificates.crt \
${NFSROOT}/usr/local/share/ca-certificates/ComodoIntermediateCertificates.crt
chroot $NFSROOT update-ca-certificates
regards
Markus Köberl
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Accessing external https repo during install
Seems the copy is done by line 1115 of usr/lib/fai/subroutines:
fcopy -SBMir /etc/apt # copy all other apt config files from the config
space
It probably should be documented, especially since docs currently state
that files under files/ are not copied automatically but require an
fcopy. Or I just missed the special treatment of sources.list.d ...
Now I have commented the repo definitions in sources.list.d/salt.list
and uncomment 'em from hooks/configure.SALT :
-8<--
#! /bin/bash
sed -i 's/^#//' $target/etc/apt/sources.list.d/salt.list
fcopy -r /etc/salt/minion.d/
$ROOTCMD apt-get update
$ROOTCMD apt-get install -y salt-minion
-8<--
Finally it seems to work as expected.
Thanks again!
Diego
Il 18/01/2024 08:23, Diego Zuccato ha scritto:
IIUC that's the same as adding 'em to the basefile. Every time an
install errors out, basefile/nfsroot must be regenerated to include
updated root certs. Error prone and time consuming.
I'm now trying to understand:
1) who is copying the whole /etc/apt/sources.list.d during
task_repository, to disable salt.list
2) initialize salt repo with a script later in the configuration phase,
when packages (including ca-certificates) are already installed
Point 1 is really unexpected and shouldn't happen by default. Currently
ruling out it gets done by one of my scripts. Just to be sure:
fcopy /etc/apt/sources
does *not* touch /etc/apt/sources.list.d/, right?
Diego
Il 17/01/2024 17:10, Markus Köberl ha scritto:
On Wednesday, 17 January 2024 16:13:02 CET Diego Zuccato wrote:
Il 17/01/2024 14:15, Carsten Aulbert ha scritto:
How can I have ca-certificates installed when the repository gets
added?
I think you could either add it into your basefile
Thought that, but would require regular maintenance, regenerating
basefile every time ca-certificates is updated.
or add it to your
hook to install ca-certificates from Debian first.
That whould be the perfect solution.
Does that make sense?
Sure it does. I just have to understand how to do it the correct way :)
First issue (that deranged me): I forgot to set SALT class for the
test-fai host, but files/etc/apt/sources.list.d/salt.list/BOOKWORM got
copied anyway... some script is fcopy-ing more than expected...
Fixed (partially) the first issue, hooks/repository.SALT (the one that
should create salt.list file...) finally got called and attempted to
install ca-certificate. But it failed. Seems I'm attempting to install
it too soon.
Uff. Work for tomorrow...
Tks for all the hints!
I have on the fai server in /etc/fai/nfsroot.conf:
FAI_DEBOOTSTRAP_OPTS="--include=ca-certificates,apt-transport-https"
and /etc/fai/nfsroot-hooks/ca-certificates:
# load deffinition of ${NFSROOT}
. /etc/fai/nfsroot.conf
mkdir -p ${NFSROOT}/usr/local/share/ca-certificates
cp /etc/fai/nfsroot-hooks/ComodoIntermediateCertificates.crt \
${NFSROOT}/usr/local/share/ca-certificates/ComodoIntermediateCertificates.crt
chroot $NFSROOT update-ca-certificates
regards
Markus Köberl
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Accessing external https repo during install
That wouldn't work, since salt.list is copied too early, before the first update, so the update fails (well, in ignores the repo but logs an error in error.log) because it can't authenticate the external repo (it misses ca-certificates, but to install ca-certificates it needs to update the repositories... circular dependency). Diego Il 18/01/2024 11:50, Andrew Ruthven ha scritto: On Wed, 2024-01-17 at 17:10 +0100, Markus Köberl wrote: FAI_DEBOOTSTRAP_OPTS="--include=ca-certificates,apt-transport-https" Hey, My approach for this kind of thing is to have a hook that install ca- certificates. Probably updatebase.SALT - or better, updatebase.CACERTIFICATES and have SALT set CACERTIFICATES Cheers, Andrew -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Making sure to partition the right disk(s)
Hello all. It's not too unusual that sometimes disks get recognized in a different order across reboots. How can I make sure I'm repartitioning the right disk and not another one containing data? I can't find any way to bind some info about HDD to "disk1" instead of "disk2". If it's not currently supported, it shouldn't be too hard to add to 20-hwdetect.sh (I can do it and share the result, if someone is interested). But if it's already supported, better to use the official method. :) -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Making sure to partition the right disk(s)
Wonderful! I'll wait for 6.2 to be out, then. For now, the use of explicit device path can be enough, but being able to tell it to "select the 2 small disks and create a RAID1" is surely way better (and handles disk replacement w/o reconfiguration). PS: looking at the source, I noticed that a partition labeled "MY-DATA" is automatically mounted to /media/data . Does it work only for boots from CD or also from network? It could be useful to store machine's static data (SSH server key, just to say one)... Diego Il 19/01/2024 09:48, Thomas Lange ha scritto: On Fri, 19 Jan 2024 09:03:57 +0100, Diego Zuccato said: > Hello all. > It's not too unusual that sometimes disks get recognized in a different > order across reboots. > How can I make sure I'm repartitioning the right disk and not another > one containing data? I can't find any way to bind some info about HDD to > "disk1" instead of "disk2". I use this script to manipulate the disklist: http://fai-project.org/download/misc/99-disklist.sh In the next FAI version 6.2 (the release may come in the next few days) there are more functions to change the order of the disks, like smallestdisk or matchdisks which can match to a certain serial number. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Making sure to partition the right disk(s)
Il 19/01/2024 10:12, Thomas Lange ha scritto: It may also work if you do a reinstallation via network and the kernel will find a partition with label MY-DATA. I guess this should also work. Seems it does not work with network boot and default config. I added: logical - 120M:preserve_lazy ext4 noauto createopts="-L MY-DATA" to disk_config and (after zapping clean the disk) it got created [*]. But it seems it doesn't get mounted (at least a custom script did not find it mounted). I don't know FAI internals enough :( Not a big issue, since it can be replaced by a few lines in a script: #!/bin/bash datadisk=/dev/disk/by-label/MY-DATA if [ -e $datadisk ]; then mkdir -p /media/data mount $datadisk /media/data src=/media/data/$(hostname -s) if [ -d $src ]; then cp -r $src/ $target/ fi fi [*] Currently wrestling with "preserved partition /dev/sda7 does not end at a cylinder boundary, parted may fail to restore the partition" messages in error.log... "disk_config" line have "align-at:1M", isn't it enough? -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Making sure to partition the right disk(s)
Tks a lot. Really interesting & useful. I prefer to keep per-host configs in their own subdir (like 94-disklist-order.d/testserver1 so I don't have to touch a working script, but that's mostly "cosmetic". Diego Il 19/01/2024 13:29, Andrew Ruthven ha scritto: On Sat, 2024-01-20 at 01:27 +1300, Andrew Ruthven wrote: On Fri, 2024-01-19 at 09:48 +0100, Thomas Lange wrote: I use this script to manipulate the disklist: http://fai-project.org/download/misc/99-disklist.sh Attached is the script which I wrote to do this. It goes in the class directory. Some of our servers have NVMe drives that should be used for operating system disks, which is why they can be skipped. Although I see a stale comment in there now about the NVMe disks. Ah well. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Define sda as the smallest disk
I think there's a bug (well, a missing piece) in 99-disklist.sh: egrep
ignores SCSI disk, considering just ATA and NVME.
If it can be useful, I also modified the script to look into
99-disklist.d for hostname-specific configs (I prefer having separate
files instead of embedding in a bigger one):
-8<-- 99-disklist -8<--
#! /bin/bash
mydisks() {
find $* -type l -printf "%f %l\n" | grep -Pv
'^md|-part\d|^wwn-|^nvme-eui|^nvme-nvme' | egrep '^scsi|^ata|^nvme' |
sed -e 's#.*/##g'| tr '\n' ' '
}
# This is really important, because we use shell globbing for creating
the list of disks
cd /dev/disk/by-id || echo Cannot get disk information
filter='nvme*'
if [ -f $0.d/$HOSTNAME ] ; then
# Source host-specific list.
# Can define a new list or just override filter
. $0.d/$HOSTNAME
fi
if [ -z $newlist ]; then
newlist=$(mydisks $filter* )
fi
if [ -n "$newlist" ]; then
echo New disklist: $newlist
echo disklist=\"$newlist\" >> $LOGDIR/additional.var
fi
-8<--
Note the missing .sh extension.
The 99-disklist.d/$HOSTNAME file is just:
-8<--
#!/bin/bash
filter='scsi-*'
# or
#newlist='sda'
-8<--
HiH.
Diego
Il 31/01/2024 13:49, Thomas Lange ha scritto:
Hi all,
that is the example how to change the shell variable $disklist:
https://fai-project.org/download/misc/99-disklist.sh
Create the script class/99-disklist.sh in your config space (/s/rv/fai/config)
These are the imprtant lines:
if [ -n "$newlist" ]; then
echo New disklist: $newlist
echo disklist=\"$newlist\" >> $LOGDIR/additional.var
fi
This script writes the new valuespf disklist to
$LOGDIR/additional.var. Then FAI will parse it and sets the new value
for disklist before calling setup-storage.
regards Thomas
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Define sda as the smallest disk
Just a followup.
My script had a serious bug (plus a typo), too: since it gets sourced,
$0 is not what I expected :( After many failed atttempts finally I
noticed it and patched.
What I just tested:
-8<--
root@fai:/srv/fai/config/class# cat 99-disklist.sh
#! /bin/bash
mydisks() {
find $* -type l -printf "%f %l\n" | grep -Pv
'^md|-part\d|^wwn-|^nvme-eui|^nvme-nvme' | egrep '^scsi|^ata|^nvme' |
sed -e 's#.*/##g'| tr '\n' ' '
}
# This is really important, because we use shell globbing for creating
the list of disks
cd /dev/disk/by-id || echo Cannot get disk information
filter='nvme*'
# can not use $0: script is sourced, not run explicitly !
datadir=$FAI/class/99-disklist.d
echo " Testing $datadir/$HOSTNAME"
if [ -f $datadir/$HOSTNAME ] ; then
# Source host-specific list. Can define a new list or override filter
. $datadir/$HOSTNAME
fi
if [ -z $newlist ]; then
echo "newlist is empty, filter=$filter"
newlist=$(mydisks $filter )
fi
if [ -n "$newlist" ]; then
echo New disklist in 99-disklist.sh: $newlist
echo "disklist=\"$newlist\" # $0" >> $LOGDIR/additional.var
fi
-8<--
And 99-disklist.d/fast00 (the host I'm installing) contains:
-8<--
#!/bin/bash
#filter='scsi-*'
#newlist='sdt '
. /usr/lib/fai/subroutines
newlist=$(smallestdisk)
-8<--
Hope it can be useful for others.
Diego
Il 22/02/2024 09:02, Diego Zuccato ha scritto:
I think there's a bug (well, a missing piece) in 99-disklist.sh: egrep
ignores SCSI disk, considering just ATA and NVME.
If it can be useful, I also modified the script to look into
99-disklist.d for hostname-specific configs (I prefer having separate
files instead of embedding in a bigger one):
-8<-- 99-disklist -8<--
#! /bin/bash
mydisks() {
find $* -type l -printf "%f %l\n" | grep -Pv
'^md|-part\d|^wwn-|^nvme-eui|^nvme-nvme' | egrep '^scsi|^ata|^nvme' |
sed -e 's#.*/##g'| tr '\n' ' '
}
# This is really important, because we use shell globbing for creating
the list of disks
cd /dev/disk/by-id || echo Cannot get disk information
filter='nvme*'
if [ -f $0.d/$HOSTNAME ] ; then
# Source host-specific list.
# Can define a new list or just override filter
. $0.d/$HOSTNAME
fi
if [ -z $newlist ]; then
newlist=$(mydisks $filter* )
fi
if [ -n "$newlist" ]; then
echo New disklist: $newlist
echo disklist=\"$newlist\" >> $LOGDIR/additional.var
fi
-8<--
Note the missing .sh extension.
The 99-disklist.d/$HOSTNAME file is just:
-8<--
#!/bin/bash
filter='scsi-*'
# or
#newlist='sda'
-8<--
HiH.
Diego
Il 31/01/2024 13:49, Thomas Lange ha scritto:
Hi all,
that is the example how to change the shell variable $disklist:
https://fai-project.org/download/misc/99-disklist.sh
Create the script class/99-disklist.sh in your config space
(/s/rv/fai/config)
These are the imprtant lines:
if [ -n "$newlist" ]; then
echo New disklist: $newlist
echo disklist=\"$newlist\" >> $LOGDIR/additional.var
fi
This script writes the new valuespf disklist to
$LOGDIR/additional.var. Then FAI will parse it and sets the new value
for disklist before calling setup-storage.
regards Thomas
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Making sure to partition the right disk(s)
Just found another issue: if the MY-DATA partition is on the target disk, reinstall errors out at partitioning because it finds that partition in use. Error happens even if the partition is correctly flagged preserve_lazy :( Enough tampering for today. BYtE, Diego Il 19/01/2024 15:44, Thomas Lange ha scritto: On Fri, 19 Jan 2024 15:33:02 +0100, Diego Zuccato said: > But it seems it doesn't get mounted (at least a custom script did not > find it mounted). I don't know FAI internals enough :( This mounting of a partition labeled MY-DATA will only work from FAI 6.2, which is not yet released. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: simple question
I'm using this /srv/fai/config/class/99-disklist.sh based on the one
linked by Thomas:
-8<--
#! /bin/bash
mydisks() {
find $* -type l -printf "%f %l\n" | grep -Pv
'^md|-part\d|^wwn-|^nvme-eui|^nvme-nvme' | egrep '^scsi|^ata|^nvme' |
sed -e 's#.*/##g'| tr '\n' ' '
}
# This is really important, because we use shell globbing for creating
the list of disks
cd /dev/disk/by-id || echo Cannot get disk information
filter='nvme*'
# can not use $0: script is sourced, not run explicitly !
datadir=$FAI/class/99-disklist.d
echo " Testing $datadir/$HOSTNAME"
if [ -f $datadir/$HOSTNAME ] ; then
# Source host-specific list. Can define a new list or override filter
. $datadir/$HOSTNAME
fi
if [ -z $newlist ]; then
echo "newlist is empty, filter=$filter"
newlist=$(mydisks $filter )
fi
if [ -n "$newlist" ]; then
echo New disklist in 99-disklist.sh: $newlist
echo "disklist=\"$newlist\" # $0" >> $LOGDIR/additional.var
fi
-8<--
Then I can place a per-host script under
/srv/fai/config/class/99-disklist.d/hostname to modify the list for
'hostname':
-8<--
#!/bin/bash
. /usr/lib/fai/subroutines
newlist=$(smallestdisk)
-8<--
I'll probably extend it a bit to also look for classes.
HIH
Diego
Il 02/07/2024 15:11, Harald Baumgartner ha scritto:
Hello,
hope it's the correct list.
OS ubuntu 22.04 LTS or higher
is it possible to choose automatically the smaller disk?
In my case there is a Dell Server with RAID-Controller, configured:
sda = ca. 5 TB SSD for data - during install shall be not
touched/formatted...
sdb = ca. 900 GB SAS = shall be root and OS installed
In other cases, sda = ca. 300 TB (data) and sdb ca. 500 GB => for OS.
Thanks,
Harald
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Installing CUDA with FAI
Nope, I'm using Salt for everything that is "personalization-after-installation", including CUDA install. But FAI acts as packages cache also for CUDA :) Diego Il 24/10/2024 13:48, Stephan Frank ha scritto: Hello everyone, has anybody ever successfully installed CUDA via FAI into a Debian Bookwork (or any other) installation? I have been trying to set this up for over a week now - yet no success. Regards, Stephan -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
migration to is-kea (was Re: FAI for Linux on Apple Silicon?)
Hi.
Hijacking thread for a possibly quick question: is a migration to
isc-kea planned ? isc-dhcp-server has been obsoleted, but the tests
we've done highlighted many problems with the change (at leas with isc's
upstream packages, not tested yet with Debian ones).
Tks,
Diego
Il 17/01/2025 12:00, Thomas Lange ha scritto:
Add this to dhcpd.conf:
if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00011" {
filename "boot/grub/grubnetaa64.efi";
}
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: migration to is-kea (was Re: FAI for Linux on Apple Silicon?)
Il 17/01/2025 18:46, Thomas Lange ha scritto: > Hijacking thread for a possibly quick question: is a migration to > isc-kea planned ? isc-dhcp-server has been obsoleted, but the tests No, there's no plan to use the kea DHCP server in the near future (not in the next two years I guesss). I always wanna use mature rock-solid software. Other people can deal with the problems that a new software will always have. But FAI itself does not depend on a specific DHCP server. I only provide some config examples for the isc-dhcp. You can also use dnsmasq if you like. Tks. When I see a deprecation notice I always worry, but it seems isc-kea is not yet ready for prime time. I'll have a look at dnsmask too. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Error partitioning with RAID
Hello all.
I'm having issues trying to create a layout from scratch.
I used the RAID example to write this:
-8<--
disk_config disk1 bootable:1 preserve_lazy:3,6
primary - 200M- -
primary swap2G - -
primary - 120M-122M - -
logical - 15G - -
logical - 10G - -
logical - 1G- - -
disk_config disk2 sameas:disk1
disk_config disk3 sameas:disk1
disk_config raid fstabkey:uuid preserve_lazy:3,6
raid1 /boot disk1.1,disk2.1,disk3.1 ext4
rw,noatime,errors=remount-ro
raid1 - disk1.3,disk2.3,disk3.3 ext4 noauto createopts="-L
PERMDATA"
raid5 / disk1.5,disk2.5,disk3.5 ext4
rw,noatime,errors=remount-ro
raid1 /var/lib/mysql disk1.6,disk2.6,disk3.6 ext4
rw,noatime,nosuid,nodev,rw
raid5 /home disk1.7,disk2.7,disk3.7 ext4 rw,noatime,nosuid,nodev,rw
-8<--
The first issue is that I have yet to grasp the difference between
preserve_reinstall and preserve_lazy. But at least when there's no
partition table and flag_initial=1 it seems preserve_reinstall errors
out (cause it can't read the partition table) while preserve_lazy allows
to continue.
But now it errors out:
-8<--
INTERNAL ERROR in setup-storage:
filesystem is undefined
Please report this error to the Debian Bug Tracking System.
at /usr/share/fai/setup-storage/Commands.pm line 57.
FAI::build_mkfs_commands("/dev/md6", HASH(0x555682c295d0)) called at
/usr/share/fai/setup-storage/commands.pm line 531
FAI::build_raid_commands called at /usr/sbin/setup-storage line 209
-8<--
Any hints?
Tks.
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Error partitioning with RAID
Something now works, but seems the problem is with the preservation of
RAID volumes: if I remove the "preserve_lazy:3,6" from RAID config, the
system installs. But overwrites partitions with data I'd need to keep.
I also tried moving preserve_lazy to volume lines like
raid1 - disk1.3,disk2.3,disk3.3:preserve_lazy ext4 noauto createopts="-L
PERMDATA"
but it fails.
Any help appreciated. Nobody have had to preserve a RAID volume between
installs and can share an example?
TIA
Diego
Il 04/03/2025 07:12, Diego Zuccato ha scritto:
Hello all.
I'm having issues trying to create a layout from scratch.
I used the RAID example to write this:
-8<--
disk_config disk1 bootable:1 preserve_lazy:3,6
primary - 200M - -
primary swap 2G - -
primary - 120M-122M - -
logical - 15G - -
logical - 10G - -
logical - 1G- - -
disk_config disk2 sameas:disk1
disk_config disk3 sameas:disk1
disk_config raid fstabkey:uuid preserve_lazy:3,6
raid1 /boot disk1.1,disk2.1,disk3.1 ext4
rw,noatime,errors=remount-ro
raid1 - disk1.3,disk2.3,disk3.3 ext4 noauto createopts="-L
PERMDATA"
raid5 / disk1.5,disk2.5,disk3.5 ext4
rw,noatime,errors=remount-ro
raid1 /var/lib/mysql disk1.6,disk2.6,disk3.6 ext4
rw,noatime,nosuid,nodev,rw
raid5 /home disk1.7,disk2.7,disk3.7 ext4
rw,noatime,nosuid,nodev,rw
-8<--
The first issue is that I have yet to grasp the difference between
preserve_reinstall and preserve_lazy. But at least when there's no
partition table and flag_initial=1 it seems preserve_reinstall errors
out (cause it can't read the partition table) while preserve_lazy allows
to continue.
But now it errors out:
-8<--
INTERNAL ERROR in setup-storage:
filesystem is undefined
Please report this error to the Debian Bug Tracking System.
at /usr/share/fai/setup-storage/Commands.pm line 57.
FAI::build_mkfs_commands("/dev/md6", HASH(0x555682c295d0)) called
at /usr/share/fai/setup-storage/commands.pm line 531
FAI::build_raid_commands called at /usr/sbin/setup-storage line 209
-8<--
Any hints?
Tks.
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Error partitioning with RAID
Il 05/03/2025 11:59, Thomas Lange ha scritto: I never tried such a configuration and I don't know if this can work. Didn't think that having to keep some RAID volumes across reinstalls would be so problematic :) Should be quite a common scenario :) Can you please check if the RAID is already assembled before setup-storage starts? I'm not sure if setup-storage can handle an already running raid, or if it wan't to detect the partitions and the reassembles the RAID itself. The RAID is not assembled, or at least when setup-storage aborts install there is no /proc/mdstat . There's a kernel cmdline option for dracut, which boots the FAI environment (option callled rd.auto=1) which can enable building the RAIDs. Currently not using it. We're still doing some tests, but this config seems to work, at least if disks are wiped when booting with 'initial' flag: -8<-- disk_config disk1 bootable:1 align-at:1M preserve_lazy:3,6,7 primary - 200M - - primary swap2G swap- primary - 120M-128M- - logical - 15G - - logical - 500G - - logical - 1G- - - disk_config disk2 sameas:disk1 disk_config disk3 sameas:disk1 # preserve_lazy references md devices only, starting from md0 ! disk_config raid fstabkey:uuid preserve_lazy:1,3,4 raid1 /boot disk1.1,disk2.1,disk3.1 ext4 rw,noatime,errors=remount-ro mdcreateopts="--metadata=0.90" raid1 - disk1.3,disk2.3,disk3.3 ext4 noauto,noatime createopts="-L PERMDATA" raid5 / disk1.5,disk2.5,disk3.5 ext4 rw,noatime,errors=remount-ro raid1 /var/lib/mysql disk1.6,disk2.6,disk3.6 ext4 rw,noatime,nosuid,nodev raid5 /home disk1.7,disk2.7,disk3.7 ext4 rw,noatime,nosuid,nodev -8<-- As noted in the comment, preserve_lazy references different objects (partitions in the first disk_config, md devices in the second). The worst thing is that it seems it's not allowed to specify preserve_lazy after the names of the md device components (diskX.Y...:preserve_lazy). Some tests on a VM were OK, and preserved partitions did not get erased. I'm still not sure why I have to wipe the disks even when using flag_initial and why preserve_reinstall errors out when disks are wiped (IIUC if flag_initial=1 it shouldn't even try to read partitions info). -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Who is using the ISC DHCP server in their FAI environment?
Hi. I already tried migrating to kea and it's been quite a pain in the a$$ (quite a lot of work). Quickly returned to the default. Also, does not bode well with centralized management (Salt, in my case), unless I missed something. If ISC DHCP can't be kept, amen. It will be quite a lot of work. If needed, I can try to recover the Kea config files I was testing. Diego Il 27/05/2025 09:33, Thomas Lange ha scritto: Hi, I like to know who is using the DHCP server from ISC when doing a FAI network installation. Debian plans to remove this package for the upcoming release (trixie) and I'm not sure how much work it may cause to sysadmin to migrate to a replacement, for e.g. the new DHCPD from ISC called kea. Did anyone already migrated to kea? Do you think you can easily migrate to kea or dnsmasq or other packages? Currently FAI provides examples for the dhcpd.conf for ISC dhcp, but if Debian removes the isc dhcp package, I cannot provide a working config for the trixie release in time and FAI users have to do it on their own. I'll try to convince the Debian release team to keep ISC dhcp for the trixie release, but some people want to remove it , because it's EOL and there will be no security support by Debian any more. What's your oppinion on that? -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Initial partitionin
Hello all. I probably misunderstood something, but shouldn't passing "initial" flag make partitioner ignore existing partitions? Say I have a disk that have to be completely wiped for installation. I couldn't make FAI completely ignore existing partitions and just wipe the disk before applying disk_config. RAID and LVM onew would make the issue worse, but the problem is there also with basic partitions... TIA -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Initial partitioning
Hi Andrej. I have many different partition schemes, sometimes per-class, sometimes per-host. The last I used is: -8<-- disk_config disk1 disklabel:msdos bootable:1 fstabkey:uuid align-at:1M preserve_reinstall:6,8 primary / 12G-16Gext4 rw,noatime,errors=remount-ro logical swap2G-4G swap sw logical - 120M-122M:preserve_lazy ext4 noauto createopts="-L PERMDATA" logical /var8G-16G ext4 rw,noatime,nosuid,nodev logical /home 2G-ext4 rw,noatime,nosuid,nodev -8<-- Since the PC was a Windows machine, the disk was already partitioned with a completely different scheme and setup_storage errored out. Might be related to preserve_* options (still couldn't grasp the difference between 'reinstall' and 'lazy') but what I need is just that those partitions are left untouched unless I tell FAI it's the first install. Diego Il 29/05/2025 09:26, andrej via linux-fai ha scritto: > Buongiorno Diego, > > How exactly are you using FAI, how have you modified your original > config files? I have no problems w/ reinstalls (blasting away/ > reinstalling machines, whether or not they have pre-existing > partitions). > > Ciao, > Andrej -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Initial partitioning
Tks! Seems partition.WIPEDISKS is wiping too much (including data disks not marked for repartitioning and hence not in $disklist), while wipe-all considers only disks in $disklist (then why '-all'?) but does not account for LVM and MDRAID. But they're a good starting point. Neither run only when flag_initial=1 :( From where is it best to call such a script? Diego Il 02/06/2025 13:50, Thomas Lange ha scritto: Maybe some LVM, soft RAID blog are still on the disk? You can find two scripts/hooks which help removing all data from all disks on https://fai-project.org/download/misc/ -- regards Thomas -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Problem with encrypted /boot
Il 04/06/2025 12:34, Thomas Lange ha scritto: We do not have a special place for user contributed scripts, this mailing list is fine IMO. You may want to add it to the FAI wiki. It's now on the wiki: https://wiki.fai-project.org/index.php/FullDiskEncryption Tks for all the support! -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Initial partitioning
Il 03/06/2025 09:43, Thomas Lange ha scritto: > > From where is it best to call such a script? > A good place is as a hook for the partition task as seen in > partition.WIPEDISKS Urgh... Stupid me: forgot about hooks. BTW, the problem that triggered my request was an install on a machine partitioned for Windows, so neither LVM nor MDRAID were involved but FAI failed repartitioning it. Once run "wipefs -af /dev/nvme0n1" and rebooted, it worked. Diego -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Problem with encrypted /boot
Hi. I'm getting mad (also) at setting up encrypted boot. The problem is that if I setup an encrypted /boot partition (possible I'm the first trying?), the keyfile does not end up in initramfs. IIUC, I have to: - have FAI_KEEP_CRYPTKEYFILE during install (e.g. via ENCRYPTED.var) - create hooks/mountdisks.ENCRYPTED that: - setup a password to unlock GRUB loading (have to use pbkdf2 instead of Argon2) after task_partition - change KEYFILE_PATTERN in /target/etc/cryptsetup-initramfs/conf-hook to match crypttab - set UMASK=0077 in /target/etc/initramfs-tools/conf.d/restrictperms.conf *BUT* /etc/crypttab contains a reference to /tmp/fai/crypt_dev_sda2 that does not exist anymore when the system is booting! (/tmp is emptied at every boot, the keyfile is in /var/log/fai/$(hostname -s)/last/): should I edit it in my mountdisks.ENCRYPTED script or is there a better (or more correct) way? TIA -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Problem with encrypted /boot
Update: it now works as expected. Might it be worthy of inclusion with
other user-contributed scripts?
class/ENCRYPTED.var :
-8<--
FAI_KEEP_CRYPTKEYFILE=1
LUKS_PASS="YourBootPassword"
-8<--
hooks/configure.ENCRYPTED :
-8<--
#!/bin/sh
# Requires ENCRYPTED.var for setup
srctab="${LOGDIR}/crypttab"
tmptab="${LOGDIR}/crypttab.tmp"
mkdir /target/etc/keys
chmod 0700 /target/etc/keys
echo > $tmptab
while read name dev keyfile x; do
yes "${LUKS_PASS}" | cryptsetup luksAddKey --key-slot=9
--pbkdf=pbkdf2 --key-file ${keyfile} ${dev}
newkeyfile="$(echo $keyfile | sed "s:${LOGDIR}:/etc/keys:")"
mv ${keyfile} /target/${newkeyfile}
chmod 0600 /target/${newkeyfile}
echo "${name} ${dev} ${newkeyfile} $x" >> $tmptab
done < $srctab
mv $tmptab $srctab
cp $srctab /target/etc/crypttab
echo "KEYFILE_PATTERN=/etc/keys/*" >>
/target/etc/cryptsetup-initramfs/conf-hook
echo "UMASK=0077" > /target/etc/initramfs-tools/conf.d/restrictperms.conf
$ROOTCMD update-initramfs -u
-8<--
The early phase of the boot asks for the password, then the GRUB menu
lets me select what to load. Since the key is now in initrd, there's no
need to ask the pass again.
CAVEAT: since the key is in the initrd, it remains in memory during
suspend. Might be a problem for some scenarios.
Diego
Il 04/06/2025 09:21, Diego Zuccato ha scritto:
Hi.
I'm getting mad (also) at setting up encrypted boot.
The problem is that if I setup an encrypted /boot partition (possible
I'm the first trying?), the keyfile does not end up in initramfs.
IIUC, I have to:
- have FAI_KEEP_CRYPTKEYFILE during install (e.g. via ENCRYPTED.var)
- create hooks/mountdisks.ENCRYPTED that:
- setup a password to unlock GRUB loading (have to use pbkdf2 instead
of Argon2) after task_partition
- change KEYFILE_PATTERN in /target/etc/cryptsetup-initramfs/conf-
hook to match crypttab
- set UMASK=0077 in /target/etc/initramfs-tools/conf.d/
restrictperms.conf
*BUT* /etc/crypttab contains a reference to /tmp/fai/crypt_dev_sda2 that
does not exist anymore when the system is booting! (/tmp is emptied at
every boot, the keyfile is in /var/log/fai/$(hostname -s)/last/): should
I edit it in my mountdisks.ENCRYPTED script or is there a better (or
more correct) way?
TIA
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
Re: Problem with encrypted /boot
Wiki is OK for me, but I'm not (yet) registered. I could format it as a howto. Diego Il 04/06/2025 12:34, Thomas Lange ha scritto: On Wed, 4 Jun 2025 12:27:27 +0200, Diego Zuccato said: > Update: it now works as expected. Might it be worthy of inclusion with > other user-contributed scripts? We do not have a special place for user contributed scripts, this mailing list is fine IMO. You may want to add it to the FAI wiki. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Howto include 3rd party repository .gpg keys using fai-cd?
Using trusted.gpg is deprecated. Just use another dir and add to the list files a "signed-by" option. Diego Il 17/07/25 06:53, Scott Ferguson ha scritto: On 14/7/25 14:00, Scott Ferguson wrote: Debian Bookworm fai-server running latest fai-project packages, creating installation .iso images for Debian Bookworm clients. I have put the .gpg keys in $NFSROOT/etc/apt/trusted.gpg.d (per 2010 instructions on this mailing list as I failed to find another guide) "sudo chroot /srv/fai/nfsroot apt-key list" shows them there. However when I create the .iso image only the default .gpg keys are there (not the ones I added). I used: sudo fai-make-nfsroot -fs cl=DEBIAN,DHCPC,DEMO,FAIBASE,BOOKWORM,ONE,BACKPORTS,SSH_SERVER,STANDARD,NONFREE,RECOMMENDS,FAIME,GRUB_PC,GRUB_EFI,AMD64 sudo fai-mirror -C /etc/fai -m1 -c$cl /srv/fai/mirror sudo fai-cd -C /etc/fai -g grub.cfg.install-only -m/srv/fai/mirror /media/host/test.iso Kind regards Some more information:- root@fai:/srv/fai/nfsroot/etc/apt/trusted.gpg.d# ls -al total 124 drwxr-xr-x 2 root root 4096 Jul 12 22:18 . drwxr-xr-x 8 root root 4096 Jul 12 17:29 .. -rw-r--r-- 1 root root 2484 Mar 25 02:22 brave-browser-archive-keyring.gpg -rw-r--r-- 1 root root 11861 Apr 10 09:04 debian-archive-bookworm-automatic.asc -rw-r--r-- 1 root root 11873 Apr 10 09:04 debian-archive-bookworm-security-automatic.asc -rw-r--r-- 1 root root 461 Apr 10 09:04 debian-archive-bookworm-stable.asc -rw-r--r-- 1 root root 11861 Apr 10 09:04 debian-archive-bullseye-automatic.asc -rw-r--r-- 1 root root 11873 Apr 10 09:04 debian-archive-bullseye-security-automatic.asc -rw-r--r-- 1 root root 3403 Apr 10 09:04 debian-archive-bullseye-stable.asc -rw-r--r-- 1 root root 11861 Apr 10 09:04 debian-archive-trixie-automatic.asc -rw-r--r-- 1 root root 11873 Apr 10 09:04 debian-archive-trixie-security-automatic.asc -rw-r--r-- 1 root root 1384 Apr 10 09:04 debian-archive-trixie-stable.asc -rw-r--r-- 1 root root 2824 May 17 07:01 fai-project.gpg -rw-r--r-- 1 root root 12775 Jul 12 07:36 google-chrome.gpg -rw-r--r-- 1 root root 2223 Aug 12 2024 signal-desktop-keyring.gpg -rw-r--r-- 1 root root 2288 May 16 14:14 tailscale-archive-keyring.gpg On the machine built from the generated .iso root@t490s:/etc/apt/trusted.gpg.d# ls -al total 96 drwxr-xr-x 2 root root 4096 Jul 17 14:17 . drwxr-xr-x 9 root root 4096 Jul 17 14:21 .. -rw-r--r-- 1 root root 11861 Apr 10 09:04 debian-archive-bookworm-automatic.asc -rw-r--r-- 1 root root 11873 Apr 10 09:04 debian-archive-bookworm-security-automatic.asc -rw-r--r-- 1 root root 461 Apr 10 09:04 debian-archive-bookworm-stable.asc -rw-r--r-- 1 root root 11861 Apr 10 09:04 debian-archive-bullseye-automatic.asc -rw-r--r-- 1 root root 11873 Apr 10 09:04 debian-archive-bullseye-security-automatic.asc -rw-r--r-- 1 root root 3403 Apr 10 09:04 debian-archive-bullseye-stable.asc -rw-r--r-- 1 root root 11861 Apr 10 09:04 debian-archive-trixie-automatic.asc -rw-r--r-- 1 root root 11873 Apr 10 09:04 debian-archive-trixie-security-automatic.asc -rw-r--r-- 1 root root 1384 Apr 10 09:04 debian-archive-trixie-stable.asc -rw-r--r-x 1 root root 2824 Nov 8 2019 DEBIAN.gpg I understand how the DEBIAN.gpg key gets there, and 'could' include my three third-party keys by creating a new class for each of the keys and then including those keys in $FAI_CONFIGDIR/package_config e.g. classes SIGNAL, TAILSCALE, GCHROME and SIGNAL.gpg, TAILSCALE.gpg and GCHROME.gpg - which 'seems' right, but doesn't explain by fai-project.gpg ends up in $NFSROOT/etc/apt/trusted.gpg in the first place even though it doesn't wind up in the created .iso. I suspect that is just lint from earlier versions of fai, but would appreciate a more educated opinion. I am using fai-quickstart 6.4.1 Kind regards
Re: Howto include 3rd party repository .gpg keys using fai-cd?
Il 29/07/25 19:28, Christopher Huhn ha scritto: Hi, Am 29.07.25 um 14:56 schrieb Diego Zuccato: Using trusted.gpg is deprecated. Hm, dut deprecation means "it will probably stop working" and not that it's already broken, doesn't it? I'm not sure of the current deprecation stage: I'm using the signed-by since I remember adding extra repositories... Just use another dir and add to the list files a "signed-by" option. Is this already supported by FAI in the way package_config/CLASS.asc is? It's supported as long as you create the files and fcopy 'em from a script. There's probably a better way, but this one works :) If I'm not mistaken apt has become very picky about the extension of the files in /etc/apt/trusted.gpg.d: * Binary keys must be named *.gpg * ASCII-armored keys must be named *.asc Probably you are dealing with the latter? Nope. I have, for example, /srv/fai/config/files/etc/apt/keyrings/salt-archive.keyring.gpg/SALT with the binary key and the installed salt.list file is: deb [signed-by=/etc/apt/keyrings/salt-archive-keyring.gpg arch=amd64] https://packages.broadcom.com/artifactory/saltproject-deb/ stable main The apt-secure man page has more details. IMHO the best way forward is to migrate the sources.list to the DEB822 format with the GPG keys inline instead of a separate file. The sources.list man page contains an example in the Signed-By section. I'll have to have a look at it. IMVHO the new format is harder to script and does not offer much more, just a step towards RedHat formats :( Diego
