Shim and secure boot status, leading up to bookworm

[Steve McIntyre]

Hey all!

Here's a status update and plans for SB and shim. If any of this is
unclear or you have doubts, please say!

We currently have *signed* shim *15.4* packages in the archive, for
all of buster, bullseye, bookworm and sid. That works OK at the
moment, but is getting old (July 2021) and needs updating soonish.

I uploaded shim *15.6* in July 2022 and we attempted to get that
signed too. Reviews were positive, but due to process problems around
Microsoft uploads and then a long delay on getting a needed EV
certificate renewed we never managed to get that signed. :-(

The MS and cert issues are now both resolved, and I'm now working on a
shim *15.7* upload. There's a little more work and testing to do, but
I'm not far off. Yay?

However, there are a couple of caveats to this...

SBAT update
---

The new shim build will need to block SB execution of older grub
builds (anything with an SBAT level for grub.debian < 4). The oldest
builds that will continue to work are:

 * 2.06-6 (unstable/bookworm)
 * 2.06-3~deb11u5 (bullseye)
 * 2.06-3~deb10u3 (buster)

This is hopefully not unexpected, but I'm sharing here to be 100%
clear. I'm planning on doing shim 15.7 builds for bullseye and buster
again, so these all matter here.

NX
--

At the end of November 2022 (while unable to get anything signed) we
passed a deadline; new shims since that point must be built with NX
support enabled, and flagged as such. This extra hardening should
improve security more, so it's not a bad thing in general.

*However*, it does have consequences - once shim is loaded by UEFI
firmware and started with NX enabled, all the UEFI binaries downstream
of it *also* have to support NX as well. Patches for grub and linux
are under discussion at the moment, but AFAIK not yet released; I need
to check on the status of fwupd-efi too.

What does this mean for us?

 * Older machines with older firmware will continue to work just fine.

 * New-enough machines with firmware that enables NX will fail to boot
   until we get full NX support through our boot chain. :-( There's a
   mitigating factor here: *such* new machines may already reject our
   older signed binaries anyway.

We're stuck in a bad situation here I'm afraid; I think the only
sensible way is forward, applying NX patches as soon as they're
ready.

Thoughts?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Yes, of course duct tape works in a near-vacuum. Duct tape works
 anywhere. Duct tape is magic and should be worshipped."
   -― Andy Weir, "The Martian"


signature.asc
Description: PGP signature

Re: Shim and secure boot status, leading up to bookworm

[Antonio Terceiro]

On Wed, Jan 25, 2023 at 06:11:45PM +, Steve McIntyre wrote:
> Hey all!
> 
> Here's a status update and plans for SB and shim. If any of this is
> unclear or you have doubts, please say!
> 
> We currently have *signed* shim *15.4* packages in the archive, for
> all of buster, bullseye, bookworm and sid. That works OK at the
> moment, but is getting old (July 2021) and needs updating soonish.
> 
> I uploaded shim *15.6* in July 2022 and we attempted to get that
> signed too. Reviews were positive, but due to process problems around
> Microsoft uploads and then a long delay on getting a needed EV
> certificate renewed we never managed to get that signed. :-(
> 
> The MS and cert issues are now both resolved, and I'm now working on a
> shim *15.7* upload. There's a little more work and testing to do, but
> I'm not far off. Yay?

Have the issues with arm64 been fixed? Will this release provide a
signed arm64 shim?


signature.asc
Description: PGP signature

Bug#1029543: hw-detect: clarify use cases about searching for firmware packages on external media

[Cyril Brulebois]

Hi,

Cyril Brulebois  (2023-01-24):
> I'm also not sure what our plan for e.g. ARM images (concatenateable
> images) would be; I don't think we'll pull any firmware packages during a
> d-i build, so they wouldn't end up under [2], but maybe it'd be feasible
> to just concatenate some file/image containing non-free-firmware packages
> (along with the associated metadata if relevant) as published by debian-cd
> or something? Or would those concatenateable images need to load firmware
> from external media anyway?
> 
>  2. 
> https://deb.debian.org/debian/dists/bookworm/main/installer-arm64/current/images/netboot/SD-card-images/
> 
> In any case, we could probably try and accommodate architecture-specific
> issues… by wrapping required calls inside proper conditions, so that
> special needs (on release archs or non-release archs) don't negatively
> impact regular users.
> 
> 
> In the meanwhile, I'll disable both mountmedia calls in the upcoming
> upload:
>  - to reduce delays when official installation images are just
>self-sufficient (that's why we had that GR in the first place!);
>  - to gather feedback/complaints from users for which those calls are
>actually important, so that we can better assess the needs, and the
>possible shortcomings in the historical implementation.

I've thought about this a little more, with the following in mind:
netboot in general and netboot SD card images in particular.


# netboot

Netboot images are built by debian-installer, and d-i builds will
certainly not start pulling things from outside main; that being said,
it's been well known that CPIO archives can be concatenated (see
https://wiki.debian.org/DebianInstaller/NetbootFirmware), so it's
possible to take the firmware directory produced by a debian-cd build
(as found on netinst images), and append a CPIO archive of that
directory to a netboot or netboot-gtk initrd. Tested by using direct
kernel loading in virt-manager, stitching things together this way:

# pick firmware/ as prepared by debian-cd (e.g. tmp/sid/CD1):
pax -L -x sv4cpio -w firmware | pigz -9c >firmware.cpio.gz

# append to d-i's initrd.gz:
cat initrd.gz firmware.cpio.gz > initrd+firmware.gz

Then boot with:
 - Kernel path: netboot/gtk/debian-installer/amd64/linux
 - Initrd path: netboot/gtk/debian-installer/amd64/initrd+firmware.gz
 - Kernel args: vga=791 (to ensure the framebuffer gets enabled)

Beware, the initial initrd.gz is 83M, the firmware.cpio.gz is 186M! That
means I needed to bump my very usual VM's RAM rom 1G to 1.5G, maybe even
2G (depending on which d-i features get used). Things should be less crazy
once we limit debian-cd builds to picking firmware packages from one
section (non-free-firmware only, instead of contrib + non-free + n-f-f),
but that's still an important size bump.

All that ensures a /firmware is found by hw-detect, and firmware can be
detected/deployed/installed as needed, just like in the netinst case.

It shouldn't be a problem to provide a pre-built firmware.cpio.gz
archive with the relevant metadata (produced by debian-cd) in the dep11
subdirectory, for use by hw-detect. We should probably start publishing
them in a less hidden location than the existing:
  
https://cdimage.debian.org/cdimage/unofficial/non-free/firmware/bookworm/20230123/

Maybe adding a “non-free-firmware” directory directly under this
location?
  https://cdimage.debian.org/cdimage/release/current/

After all, each arch-specific directory will have non-free-firmware bits
included, so having an extra directory in the release directory shouldn't
be an issue?

And we might clean things up in:
  https://wiki.debian.org/DebianInstaller/NetbootFirmware

Maybe strongly recommend starting from the upcoming new firmware.cpio.gz
file, and only keep the relevant bits (this and that firmware package,
and relevant metadata), to keep the overhead low?


# SD card images (might also be applicable to the upcoming ChromeOS images)

Looking at how those images are built, it seems we have a predefined,
per-arch image size (150M or 200M), and gen-hd-image that's called in two
modes: firmware or partition. The former takes care of the board-specific
part (e.g. firmware.a64-olinuxino.img.gz) while the latter takes care of
the board-independent part (partition.img.gz). Both can be concatenated
with the following command as documented in the README:
  
https://deb.debian.org/debian/dists/bullseye/main/installer-arm64/current/images/netboot/SD-card-images/README.concatenateable_images

zcat firmware..img.gz partition.img.gz > complete_image.img

And that works, generating an image with a single fat16, fat32, or ext2
partition with contents merged together, because of the pre-defined image
size.

We could think about bumping the image size, but given the size of the
firmware directory (as in containing the firmware-* packages and metadata,
not the board-specific firmware files), this would seem like a waste of
space for all those not needing firm

Re: Shim and secure boot status, leading up to bookworm

[Jeremy Hall]

Hi,

When things get built, will there be a path forward for people who
might need grub modules like serial console for accessibility reasons?

Thanks

_J

On 1/25/23, Antonio Terceiro  wrote:
> On Wed, Jan 25, 2023 at 06:11:45PM +, Steve McIntyre wrote:
>> Hey all!
>>
>> Here's a status update and plans for SB and shim. If any of this is
>> unclear or you have doubts, please say!
>>
>> We currently have *signed* shim *15.4* packages in the archive, for
>> all of buster, bullseye, bookworm and sid. That works OK at the
>> moment, but is getting old (July 2021) and needs updating soonish.
>>
>> I uploaded shim *15.6* in July 2022 and we attempted to get that
>> signed too. Reviews were positive, but due to process problems around
>> Microsoft uploads and then a long delay on getting a needed EV
>> certificate renewed we never managed to get that signed. :-(
>>
>> The MS and cert issues are now both resolved, and I'm now working on a
>> shim *15.7* upload. There's a little more work and testing to do, but
>> I'm not far off. Yay?
>
> Have the issues with arm64 been fixed? Will this release provide a
> signed arm64 shim?
>

Re: Shim and secure boot status, leading up to bookworm

[Steve McIntyre]

Hey Antonio,

On Wed, Jan 25, 2023 at 04:17:50PM -0300, Antonio Terceiro wrote:
>On Wed, Jan 25, 2023 at 06:11:45PM +, Steve McIntyre wrote:
>> 
>> The MS and cert issues are now both resolved, and I'm now working on a
>> shim *15.7* upload. There's a little more work and testing to do, but
>> I'm not far off. Yay?
>
>Have the issues with arm64 been fixed? Will this release provide a
>signed arm64 shim?

We should have a signed shim for arm64, yes. I need to test end to end
again yet; I think we're still missing some arm64 SB patches for grub.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
“Rarely is anyone thanked for the work they did to prevent the
 disaster that didn’t happen.”
   -- Mikko Hypponen (https://twitter.com/mikko/)

Re: Shim and secure boot status, leading up to bookworm

[Steve McIntyre]

Hi Jeremy,

On Wed, Jan 25, 2023 at 12:40:07PM -0700, Jeremy Hall wrote:
>
>When things get built, will there be a path forward for people who
>might need grub modules like serial console for accessibility reasons?

The serial module has already been added to the signed grub binary a
while back (2.06-4). If you need anything else, please ask or file
bugs.

In the longer term, some grub upstream developers are working on
adding support for signing grub modules individually that might make
it possible for people to add more themselves. But that's not going to
happen before bookworm.

HTH!

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
“Why do people find DNS so difficult? It’s just cache invalidation and
 naming things.”
   -– Jeff Waugh (https://twitter.com/jdub)

Re: Bug#1029543: hw-detect: clarify use cases about searching for firmware packages on external media

[Diederik de Haas]

On Wednesday, 25 January 2023 20:22:33 CET Cyril Brulebois wrote:
> # SD card images (might also be applicable to the upcoming ChromeOS images)
> ...
> All of this assuming that the end results can be appended as the third
> part of the  +  +  combination!

This sounds like a nice solution for the SD card images.
I myself always use a wired connection, but I saw yesterday that someone tried 
to use d-i for RockPro64 but didn't get any output on screen and then tried 
again using their TV ... but then it needed to use a wireless connection and I 
actually had no idea how that could/should be provided during install.

Giving ppl the option to concat a firmware archive if they need it, sounds like 
an elegant solution to me :-)

signature.asc
Description: This is a digitally signed message part.

Re: Bug#1029543: hw-detect: clarify use cases about searching for firmware packages on external media

[Diederik de Haas]

On Wednesday, 25 January 2023 23:01:01 CET Diederik de Haas wrote:
> This sounds like a nice solution for the SD card images.
> I myself always use a wired connection, but I saw yesterday that someone
> tried to use d-i for RockPro64 but didn't get any output on screen and then
> tried again using their TV ... but then it needed to use a wireless
> connection and I actually had no idea how that could/should be provided
> during install.
> 
> Giving ppl the option to concat a firmware archive if they need it, sounds
> like an elegant solution to me :-)

Please ignore me. Apparently I didn't fully and/or properly understand.
The ('wireless') firmware archive is likely to be very involved to construct, 
let alone other issues (like size) it may add.

signature.asc
Description: This is a digitally signed message part.

Bug#1029543: hw-detect: clarify use cases about searching for firmware packages on external media

[Vagrant Cascadian]

On 2023-01-25, Cyril Brulebois wrote:
> Cyril Brulebois  (2023-01-24):
...
> # SD card images (might also be applicable to the upcoming ChromeOS images)
>
> Looking at how those images are built, it seems we have a predefined,
> per-arch image size (150M or 200M), and gen-hd-image that's called in two
> modes: firmware or partition. The former takes care of the board-specific
> part (e.g. firmware.a64-olinuxino.img.gz) while the latter takes care of
> the board-independent part (partition.img.gz). Both can be concatenated
> with the following command as documented in the README:
>   
> https://deb.debian.org/debian/dists/bullseye/main/installer-arm64/current/images/netboot/SD-card-images/README.concatenateable_images
>
> zcat firmware..img.gz partition.img.gz > complete_image.img
>
> And that works, generating an image with a single fat16, fat32, or ext2
> partition with contents merged together, because of the pre-defined image
> size.
>
> We could think about bumping the image size, but given the size of the
> firmware directory (as in containing the firmware-* packages and metadata,
> not the board-specific firmware files), this would seem like a waste of
> space for all those not needing firmware files (probably requiring twice
> the current size to make sure everything fits)…

Well, empty space does compress quite well, at least!

Though a larger image would take a bit longer and burn a few extra write
cycles for those who did not need to actually use the space.

I do not know that we are all that size-constrained; it is hard to come
by USB sticks or SD cards that are less that 4GB these days. So even
bumping the image size to 500MB or even 1GB does not seem unreasonable
to me... I would rather err on the side of slightly larger than
currently needed; maybe there are use-cases I am not considering?

Another option is to concatenate the firmware part directly to the
initrd image manually, though it is a bit of a cumbersome
process... soemthing like:

  gunzip partition.img
  mount -o loop partition.img /mnt
  cat kernelfirmware.cpio.gz >> /mnt/initrd.gz
  umount /mnt
  zcat firmware.BOARD.gz partition.img > complete.img

Though I realize that is quite a few extra steps for an already slightly
complicated manual two-part SD card image process... but it could
probably be scripted; it is not absurdly complicated. (presuming you
have the kernel-firmware.cpio.gz creation sorted out)

This technically only requires changing the size of the partition.img to
make more room for the kernel firmware image, but otherwise would be
identical to the current process for those who did not need to add the
firmware.

Another option would be to use a growable filesystem, and then grow it
to whatever size needed? Are FAT filesystems reasonably growable?

Thanks for all your work on this!

live well,
  vagrant


signature.asc
Description: PGP signature

Bug#1029543: hw-detect: clarify use cases about searching for firmware packages on external media

[Cyril Brulebois]

Hi Vagrant,

Thanks for your feedback!

Vagrant Cascadian  (2023-01-25):
> Well, empty space does compress quite well, at least!

:D

> Though a larger image would take a bit longer and burn a few extra
> write cycles for those who did not need to actually use the space.

I have no idea how much people care how much time is spent copying things
around, but I suppose that at least for those repeatedly copying things
around (e.g. for debugging purposes), more writes could mean accelerated
wear-and-tear?

(The only things in that area I'm familiar with is Pi images that contain
full-blown systems, which are in the 1-2G ball park already.)

> I do not know that we are all that size-constrained; it is hard to come
> by USB sticks or SD cards that are less that 4GB these days. So even
> bumping the image size to 500MB or even 1GB does not seem unreasonable
> to me... I would rather err on the side of slightly larger than
> currently needed; maybe there are use-cases I am not considering?

Yeah, I've seen you notice that you were aiming to allow for a little more
room, when you bumped sizes, rather than increasing them minimally. I'm
just wondering whether it makes sense (how often is that really useful?).

> Another option is to concatenate the firmware part directly to the
> initrd image manually, though it is a bit of a cumbersome
> process... soemthing like:
> 
>   gunzip partition.img
>   mount -o loop partition.img /mnt
>   cat kernelfirmware.cpio.gz >> /mnt/initrd.gz
>   umount /mnt
>   zcat firmware.BOARD.gz partition.img > complete.img

That looks much easier than anything we could come up with if we were
trying to concatenate 3 parts (the third one would need to be created
depending on the first one — fat or ext2). Requiring mount could be
considered a tad dangerous, but big red warnings and all that…

> Though I realize that is quite a few extra steps for an already slightly
> complicated manual two-part SD card image process... but it could
> probably be scripted; it is not absurdly complicated. (presuming you
> have the kernel-firmware.cpio.gz creation sorted out)

I'm working on it right now, trying to also reduce code duplication in
debian-cd for easier long-term maintenance. I should have some proof of
concept by tomorrow, but we need to find a better location that under
unrelease/, so it won't be deployed right away.

> This technically only requires changing the size of the partition.img to
> make more room for the kernel firmware image, but otherwise would be
> identical to the current process for those who did not need to add the
> firmware.

Yes!

> Another option would be to use a growable filesystem, and then grow it
> to whatever size needed? Are FAT filesystems reasonably growable?

Just to be sure, are we talking about something like the following?
 - cat both parts as usual to get complete_image.img (via README);
 - fallocate to make it bigger for those who want more stuff;
 - fdisk to extend the partition to the new end of disk (making sure
   not to change the beginning, firmwares are picky as you know; not
   lose the signature, etc);
 - fatresize or resize2fs to extend the filesystem;
 - mount, copy, umount;
 - hope that none of this changed tricky parts that would break the
   board-specific firmare/bootloader/config/etc.

I've used fatresize on FAT32 partitions in the past (for USB storage
on Intel-based machines, but nothing firmware/bootloader-related) quite
successfully. A quick grep through its source report this claim:

The FAT16/FAT32 non-destructive resizer.

All of this (if it works) seems reasonable enough, or at least I don't
think should be a big issue for experienced users.

But I suspect less advanced users wanting to toy with an ARM board would
probably just follow the README to prepare the installer image, and rely
on an extra USB stick to hold the firmware part (and rely on mountmedia).


But again, I have no idea how common it is for the boards we support in
the installer to require loading firmware files during the installation
process…


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#1029543: hw-detect: clarify use cases about searching for firmware packages on external media

[Vagrant Cascadian]

On 2023-01-26, Cyril Brulebois wrote:
> Vagrant Cascadian  (2023-01-25):
>> Though a larger image would take a bit longer and burn a few extra
>> write cycles for those who did not need to actually use the space.
>
> I have no idea how much people care how much time is spent copying things
> around, but I suppose that at least for those repeatedly copying things
> around (e.g. for debugging purposes), more writes could mean accelerated
> wear-and-tear?

Nor I, really... so yes, just some extra writes that might be needless
in most cases.


>> I do not know that we are all that size-constrained; it is hard to come
>> by USB sticks or SD cards that are less that 4GB these days. So even
>> bumping the image size to 500MB or even 1GB does not seem unreasonable
>> to me... I would rather err on the side of slightly larger than
>> currently needed; maybe there are use-cases I am not considering?
>
> Yeah, I've seen you notice that you were aiming to allow for a little more
> room, when you bumped sizes, rather than increasing them minimally. I'm
> just wondering whether it makes sense (how often is that really useful?).

It is mostly useful with new kernel versions where the module
names/dependencies/indirect-dependencies have changed between kernel
versions, and I haven't yet figured out which kernel modules go in which
.udeb files.  Similar for new boards, just trying to figure out which
modules are needed at all.

The main approach I use is appending all modules (or more selective
modules) from /lib/modules/VERSION/ in a cpio archive that is appended
to the debian-installer initrd.gz... then compare the lsmod results from
a booted d-i... it is not the simplest process with many iteration
cycles; any reduction in required steps is a huge benefit for that!

It does seem similar to the use-case of loading extra firmware in the
installer... appending selective firmwares or all the firmwares to the
initrd as a way to make sure it is loadable early in debian-installer.


Another use-case for more-free-space-by-default are things like
raspberry pi boot firmware, which need to be loaded from the 1st FAT
partition on the microSD. Do not recall off the top of my head how large
rpi boot firmware is. I have manually copied it onto debian-installer SD
card images in the past, possibly doing a partition/filesystem resize
dance...


>> Another option is to concatenate the firmware part directly to the
>> initrd image manually, though it is a bit of a cumbersome
>> process... soemthing like:
>> 
>>   gunzip partition.img
>>   mount -o loop partition.img /mnt
>>   cat kernelfirmware.cpio.gz >> /mnt/initrd.gz
>>   umount /mnt
>>   zcat firmware.BOARD.gz partition.img > complete.img
>
> That looks much easier than anything we could come up with if we were
> trying to concatenate 3 parts (the third one would need to be created
> depending on the first one — fat or ext2). Requiring mount could be
> considered a tad dangerous, but big red warnings and all that…

Ok, good!


>> Though I realize that is quite a few extra steps for an already slightly
>> complicated manual two-part SD card image process... but it could
>> probably be scripted; it is not absurdly complicated. (presuming you
>> have the kernel-firmware.cpio.gz creation sorted out)
>
> I'm working on it right now, trying to also reduce code duplication in
> debian-cd for easier long-term maintenance. I should have some proof of
> concept by tomorrow, but we need to find a better location that under
> unrelease/, so it won't be deployed right away.

Yeah, figured that was in-process :)


>> Another option would be to use a growable filesystem, and then grow it
>> to whatever size needed? Are FAT filesystems reasonably growable?
>
> Just to be sure, are we talking about something like the following?
>  - cat both parts as usual to get complete_image.img (via README);
>  - fallocate to make it bigger for those who want more stuff;
>  - fdisk to extend the partition to the new end of disk (making sure
>not to change the beginning, firmwares are picky as you know; not
>lose the signature, etc);
>  - fatresize or resize2fs to extend the filesystem;
>  - mount, copy, umount;
>  - hope that none of this changed tricky parts that would break the
>board-specific firmare/bootloader/config/etc.

Yeah, that is basically what I was thinking.

The biggest pain point is some boot firmware likes to be on particular
raw offsets. Some offsets are incompatible with GPT (allwinner)... some
load data from offsets that edge into 4MB (ti?) or even 16MB
(rockchip!)... I think most partitioning tools default to creating with
a 2MB offset for the first partition. Some models of raspberry pi
require the boot firmare to be on the 1st fat partition.

As long as you leave those first 16MB clear and don't use GPT
partitioning... dare I say it ... I'm not aware of other issues that one
would get tripped up on!


> I've used fatresize on FAT32 partitions in the past (for USB storage
> on Int