> On 05/08/2024 09:17, Thore Sommer wrote: > > while updating our distro for ARM from bookworm to trixie, I noticed > > that our builds using qemu user-mode are now extremely slow. A > > debootstrap went from 5min for bookworm to 38min for trixie. > > I'll assume that you use the git version of live-build.
Yes, I'm using the latest git version. > > I'm currently investigating why this happens, but before I start > > profiling QEMU, I thought I ask you where to look, as you might have > > seen the same phenomenon. Was there an change in trixie that might > > affect emulation performance (i.e. new security features by enabled > > default)? > > There has indeed been a change: > https://salsa.debian.org/live-team/live-build/-/commit/2f1acabc41414a596b5e7552166a4082f738d0e2 > changed the logic to detect whether the bootstrap phase would need to > run with the architecture of the host, or under QEMU. > The file `scripts/build/bootstrap_debootstrap` now checks whether > LB_BOOTSTRAP_QEMU_ARCHITECTURE is set to any value (i.e. if > `--bootstrap-qemu-arch` is used in `lb config`), and if set, will use a > 2-stage bootstrap, which is much slower. > > However, if you only changed the `--distribution` commandline option, > there should be no significant difference. > > Can you post your `lb config` line? lb config --distribution trixie --architectures arm64 --bootstrap-qemu-arch arm64 --bootstrap-qemu-static /usr/local/bin/qemu-aarch64 But I also observed this just with plain debootstrap: debootstrap --arch=arm64 --foreign bookworm bookworm http://httpredir.debian.org/debian/ chroot bookworm /debootstrap/debootstrap --second-stage Which completes in around 5min. debootstrap --arch=arm64 --foreign trixie trixie http://httpredir.debian.org/debian/ chroot trixie /debootstrap/debootstrap --second-stage This completes in around 38min. Note that I got those timings with the latest version of QEMU, but also happens with the ones shipped in Debian. > > I already tried with a variety QEMU versions (5.2, 8.1, latest git > > snapshot), distros (bookworm, trixie) and kernel combinations, with no > > effect and because the bookworm builds are still way faster, I assume > > something has changed in how trixie is build that makes emulation slower. > > I've recently posted a break-down of the execution times of a > cross-build for the GNOME image (trixie): > https://lists.debian.org/debian-cd/2024/08/msg00001.html Have you tried to get similar timings for bookworm? Our normal live-build bookworm based ISO builds for around 2h using user-mode emulation. Only tried to run our full trixie build once which was not complete after 5h. > > That also takes about 30 minutes for the bootstrap. > > These are my current timings: > lb config --distribution bookworm --architecture arm64 > --bootstrap-qemu-arch arm64 --bootstrap-qemu-static > /usr/bin/qemu-aarch64-static > lb bootstrap > > bookworm: 6 minutes > trixie: 38 minutes > I guess that the main cause will be debootstrap. > > You might want to try a pending MR as well. > https://salsa.debian.org/live-team/live-build/-/merge_requests/343 I ran the basic experiment also with mmdebstrap: time mmdebstrap --architecture=arm64 bookworm bookworm2 http://httpredir.debian.org/debian/ real 3m12.935s time mmdebstrap --architecture=arm64 trixie trixie2 http://httpredir.debian.org/debian/ real 5m28.142s So still slower, but not as much as before. This only accounts for 30min of the 3h+ more that the trixie builds take in comparison to the bookworm builds. I'll try to get some timings for ISOs with that MR for bookworm and trixie. Best regards Thore
