Hi, Cyril Brulebois <k...@debian.org> (2023-01-24): > I'm also not sure what our plan for e.g. ARM images (concatenateable > images) would be; I don't think we'll pull any firmware packages during a > d-i build, so they wouldn't end up under [2], but maybe it'd be feasible > to just concatenate some file/image containing non-free-firmware packages > (along with the associated metadata if relevant) as published by debian-cd > or something? Or would those concatenateable images need to load firmware > from external media anyway? > > 2. > https://deb.debian.org/debian/dists/bookworm/main/installer-arm64/current/images/netboot/SD-card-images/ > > In any case, we could probably try and accommodate architecture-specific > issues… by wrapping required calls inside proper conditions, so that > special needs (on release archs or non-release archs) don't negatively > impact regular users. > > > In the meanwhile, I'll disable both mountmedia calls in the upcoming > upload: > - to reduce delays when official installation images are just > self-sufficient (that's why we had that GR in the first place!); > - to gather feedback/complaints from users for which those calls are > actually important, so that we can better assess the needs, and the > possible shortcomings in the historical implementation.
I've thought about this a little more, with the following in mind: netboot in general and netboot SD card images in particular. # netboot Netboot images are built by debian-installer, and d-i builds will certainly not start pulling things from outside main; that being said, it's been well known that CPIO archives can be concatenated (see https://wiki.debian.org/DebianInstaller/NetbootFirmware), so it's possible to take the firmware directory produced by a debian-cd build (as found on netinst images), and append a CPIO archive of that directory to a netboot or netboot-gtk initrd. Tested by using direct kernel loading in virt-manager, stitching things together this way: # pick firmware/ as prepared by debian-cd (e.g. tmp/sid/CD1): pax -L -x sv4cpio -w firmware | pigz -9c >firmware.cpio.gz # append to d-i's initrd.gz: cat initrd.gz firmware.cpio.gz > initrd+firmware.gz Then boot with: - Kernel path: netboot/gtk/debian-installer/amd64/linux - Initrd path: netboot/gtk/debian-installer/amd64/initrd+firmware.gz - Kernel args: vga=791 (to ensure the framebuffer gets enabled) Beware, the initial initrd.gz is 83M, the firmware.cpio.gz is 186M! That means I needed to bump my very usual VM's RAM rom 1G to 1.5G, maybe even 2G (depending on which d-i features get used). Things should be less crazy once we limit debian-cd builds to picking firmware packages from one section (non-free-firmware only, instead of contrib + non-free + n-f-f), but that's still an important size bump. All that ensures a /firmware is found by hw-detect, and firmware can be detected/deployed/installed as needed, just like in the netinst case. It shouldn't be a problem to provide a pre-built firmware.cpio.gz archive with the relevant metadata (produced by debian-cd) in the dep11 subdirectory, for use by hw-detect. We should probably start publishing them in a less hidden location than the existing: https://cdimage.debian.org/cdimage/unofficial/non-free/firmware/bookworm/20230123/ Maybe adding a “non-free-firmware” directory directly under this location? https://cdimage.debian.org/cdimage/release/current/ After all, each arch-specific directory will have non-free-firmware bits included, so having an extra directory in the release directory shouldn't be an issue? And we might clean things up in: https://wiki.debian.org/DebianInstaller/NetbootFirmware Maybe strongly recommend starting from the upcoming new firmware.cpio.gz file, and only keep the relevant bits (this and that firmware package, and relevant metadata), to keep the overhead low? # SD card images (might also be applicable to the upcoming ChromeOS images) Looking at how those images are built, it seems we have a predefined, per-arch image size (150M or 200M), and gen-hd-image that's called in two modes: firmware or partition. The former takes care of the board-specific part (e.g. firmware.a64-olinuxino.img.gz) while the latter takes care of the board-independent part (partition.img.gz). Both can be concatenated with the following command as documented in the README: https://deb.debian.org/debian/dists/bullseye/main/installer-arm64/current/images/netboot/SD-card-images/README.concatenateable_images zcat firmware.<board_name>.img.gz partition.img.gz > complete_image.img And that works, generating an image with a single fat16, fat32, or ext2 partition with contents merged together, because of the pre-defined image size. We could think about bumping the image size, but given the size of the firmware directory (as in containing the firmware-* packages and metadata, not the board-specific firmware files), this would seem like a waste of space for all those not needing firmware files (probably requiring twice the current size to make sure everything fits)… I'm not sure how often boards installed via SD card images require firmware packages, and how many of them, maybe it would be feasible to include instructions or even a script that only extracts this and that firmware package (and metadata) from the upcoming new firmware.cpio.gz, so that it can be appended after partition.img.gz, provided there's enough space within the defined image size? But I think it would require at least detecting the underlying partition type, determining the right numbers, and calling either mcopy or genext2fs with the right parameters… https://sources.debian.org/src/debian-installer/20210731%2Bdeb11u7/build/util/gen-hd-image/#L348-L367 All of this assuming that the end results can be appended as the third part of the <board firmware> + <partition> + <firmware> combination! (Anticipated caveat: we would likely need to tweak filenames for some FAT variants, I haven't restricted myself to the good old 8.3 format.) # Summary For regular netboot images, existing firmware addition should keep working, even if we should produce a slightly different firmware CPIO archive (adding metadata), publish it into a more discoverable place, and update instructions accordingly. For those *not* adding firmware stuff, it might make sense to keep mountmedia calls; an easy way to find out when those might make sense would be to check if a (/cdrom)/firmware directory exists at all (it would on netinst, wouldn't on netboot, would on netboot augmented with firmware stuff). For netboot SD card images, we are space-constrained, so *maybe* adding a few selected firmware packages (and metadata for automated use by hw-detected) could work, but I suppose the general case might end up users don't add anything. Especially since that's not just about concatenating compressed initrds together, we would need some more elaborate glue… All this reinforces my impression regarding adding (/cdrom)/firmware detection. Since we're talking about an installation image with a single partition, maybe those mountmedia calls work more reliably when there's external storage plugged in anyway? Again, I don't know much about how mountmedia works, or about problems/limitations users might have encountered. I might prepare the upcoming D-I release without mountmedia at all though, that's probably the best way to get feedback about it. Cheers, -- Cyril Brulebois (k...@debian.org) <https://debamax.com/> D-I release manager -- Release team member -- Freelance Consultant
signature.asc
Description: PGP signature
