TL;DR: No progress on the social side (doc) yet, but lots of progress on the technical side (making firmware-enabled images work better)!
Kernel team, feel free to skip to @KT for a little question. I'd like to thank Ben again for the modalias thing, this made it very easy to deploy firmware packages! Cyril Brulebois <k...@debian.org> (2021-07-24): > # main-only images > > ## isenkram-based proof-of-concept […] > - The issues I fixed in isenkram-cli were around some fallback code > path, and the AppStream maintainer suggested there might have been > some issues in AppStream causing the primary code path to fail: > https://bugs.debian.org/989884#24 > The relevant update was accepted into testing since then, but I > haven't had a chance to check that hypothesis. It would be great to > know whether we actually have two working lookup implementation and > not just the fallback one. (This could mean more reliability during > a release cycle, in case one of them breaks — again.) I've tried downgrading isenkram-cli to the version before my fixes regarding the fallback code path, and it didn't find anything, so either the primary code path or appstream is still broken. I've made a note to possibly investigate this further, but this definitely doesn't block the release. > ## proposed solution > > If others agree, I suggest we add the following somewhere in the > installation guide (I don't think we should have a specific warning in > the installer; or more specifically it would be way too late to have it > be shown localized at this stage; more on that in the second part of my > mail). Heavily summarized (to give readers of this bug report an idea, > not for direct user consumption!!!): […] Given Paul's ACK, I'll turn this into a bug report against the installation guide, with a slightly better draft. > ---------------------------------------------------------------------- > > # firmware-enabled images > > ## easy, but ugly fix Simon's solution makes the fallback plan even more straightforward. I don't think we'll need it though. Read on! > ## more targeted approach: modalias information > > Ben suggested augmenting AppStream metadata (see DEP-11) of > firmware-nonfree binaries with modalias information, so that it would > be feasible to establish some hardware → firmware mapping. It's not been > included in the archive yet, see MR#19 for src:firmware-nonfree: > https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/19 This was uploaded to unstable, I'm currently waiting for the updated metadata to show up in the archive. Meanwhile, I've written a new tool for debian-cd, which Steve has merged and refined further to match other choices made in that area (something like: due to size constraints, we don't ship all firmware packages), so I'm confident either of us should be able to refine this further once we actually have metadata in place. That happened in debian-cd master, and was uploaded as debian-cd 3.1.35: https://salsa.debian.org/images-team/debian-cd/-/blob/master/tools/generate_firmware_patterns Basically, we extract the relevant bits from the DEP-11 YAML file, and generate patterns files that we ship on the firmware-enabled images: - /firmware has firmware-*.deb - /firmware/dep-11 has firmare-*.patterns Meanwhile, I've hacked d-i to include those files, built manually while the rest was being worked on, so that I could test the runtime on my test laptops. > This doesn't help us (yet) regarding Intel SOF, since that package is > (ultimately) built from src:firmware-sof, which doesn't have this > logic yet. But I don't think we should delay the release for that (it > could be listed in errata, and/or fixed in 11.1, whenever the package > is ready), and if the machine at least boots fine, including the > graphics stack, the isenkram trick above should let users install the > remaining firmware packages on their own. The isenkram-based fallback (once we have the actual doc) is still a valid way to ensure this package can get installed from within the installed system, but I've also included the relevant IDs as a special case on the debian-cd side, so that we don't have to worry about that one (it seems to show up quite often on laptops, so…): https://salsa.debian.org/images-team/debian-cd/-/blob/master/tools/generate_firmware_patterns#L135-160 > There might be other “widely spread” firmware that might benefit from > the same logic, but I won't be spending too much time thinking about > this right now; such patches can come later, and those updated package > firmwares should be pickled up automatically afterwards. > > Regarding the implementation, my current approach would be: > - have firmware-nonfree's MR merged and uploaded; DONE. > - make sure the relevant information bubbles up to the dep11/Components > files (YAML format, even if the excerpts above were XML); TODO. > - have debian-cd extract the relevant firmware package←→modalias > mapping from YAML into one or N flat files to be consumed by d-i, for > firmware-enabled images primarily (or exclusively); DONE, to be checked once the previous item is complete, and some fixups might be required until a full image build behaves like my heavily hacked netboot-gtk images (meaning purely from debian-installer.git, no debian-cd involved). > - patch d-i to consume that or those files to queue up as many firmware > packages as required. DONE. I've patched hw-detect to use that information in addition to the usual grepping of dmesg to find missing firmware, and reused the logic that was there already to enable non-free and contrib (in that order, which I didn't think it was reasonable to fix at this stage of the freeze[1]), and install firmware packages inside /target (the installed system, as opposed to the installer's context). 1. https://salsa.debian.org/installer-team/apt-setup/-/commit/ba70d27d150c5fd869653082ee9180410346436b This seems to work great on both test laptops so far, and after installation, isenkram-autoinstall-firmware basically does nothing, since d-i took care of approximately everything. I have still a slight discrepancy for some BT firmware files that aren't seen in dmesg but that are apparently documented in modinfo, which isenkram spots, leading to an extra firmware-iwlwifi on one of my test laptops. I haven't tested full desktop installations, so I'm not sure whether that's actually needed, but that seems pretty minor at this point. If users are actually wondering about that, they can still follow the isenkram-based fallback (once we have the actual doc). Therefore, I'm considering this very minor, keeping a note about it, but I won't look into it before 11.0. I've also fixed and tested #973733 since one of those laptops was playing tricks regarding actual module names, making the reloading of modules after firmware installation… insufficient. Plus fixed a few other things so that logs are somewhat cleaner: https://salsa.debian.org/installer-team/hw-detect/-/commits/master @KT: I haven't uploaded it since I'm seeing iwlwifi requesting what appears to be a debug-only firmware (iwl-debug-yoyo.bin), that's even requested via an aptly-named firmware_request_nowarn() function in linux.git; I'd like to check with the kernel maintainers whether it's expected for this message to show up in dmesg still (given the name and comments in linux.git, I'm not sure it should). In any cases, it's slightly sad to pause and prompt users from firmware files we don't ship anywhere (AFAICT), that they don't actually need. Therefore, I'm tempted to blacklist this firmware file in hw-detect (i.e. pretend it was never requested). > Note: it's likely that at least some firmware packages are going to be > installed rather early in the installation process; network drivers > (wired and/or wireless ones) are likely going to be probed early, and > have generated the relevant messages in dmesg. That's exactly what happens, we have: - check-missing-firmware: dealing with hot-deploying firmware files and reloading modules. - install-firmware: dealing with the installation of firmware packages into /target (carrying over those seen by check-missing-firmware, plus the new bunch thanks to modalias information). > I have still to familiarize myself with the whole (existing) > “automatic installation of firmware packages when running on a > firmware-enabled image” logic though, to see where the patches would > be best suited. All done. As a bonus, I've (re)discovered that netcfg doesn't propagate WPA settings to the installed system when NM isn't installed there. Considering this not-a-new-problem©®™, and that shouldn't affect end users too much given they're likely going to install a desktop environment (others might know about wpa_supplicant); I'd like to investigate this at some later point, and hopefully get a fix merged into 11.1, and 10.n if I'm successful. > ---------------------------------------------------------------------- > > # final thought > > Circling back to the main-only images, having that hardware→firmware > lookup would let us have a chance to mention that “this and that and > that firmware package might be needed on your system”, adding a pointer > to the “isenkram procedure” of the installation guide. If that'd be > desired, the relevant mapping file would need to end up not only in > firmware-enabled images, and we would need new strings, new > translations, and that's almost certainly only achievable during the > next release cycle. > > In any cases, I'm happy to start with something specific to the > firmware-enabled images anyway, so that the possible fallouts are > contained within a rather known and well-defined area. We should have > enough time to implement that for 11.0, but we don't have *plenty*. We went exactly that route with Steve: changing things under /firmware only, so that there can't (or really shouldn't) be any fallouts on main-only images. The rest can wait until bookworm (if we even want to implement such a thing). Cheers, -- Cyril Brulebois (k...@debian.org) <https://debamax.com/> D-I release manager -- Release team member -- Freelance Consultant
signature.asc
Description: PGP signature