This is very interesting, thank you. I hope this ends up as a research paper.
jhigh On Thu, May 2, 2024, 15:53 Jan-Benedict Glaw <jbg...@lug-owl.de> wrote: > Hi! > > Some time ago, I started to look into NetBSD's success after reaching > reproducibility for at least two ports. Being both a VAX enthusiast > and somebody running CI builds for NetBSD current, I used that > infrastructure to work on reproducibility for VAX, as a cross-build > target building on a Linux host. That was extended to build all > port/arch combinations listed in src/build.sh . After a pile of > additional patches, all buildable port/arch combinations could be > build reproducible on Linux, a few patches later also on NetBSD. These > days, I'm chasing differences between Linux- and NetBSD-built release > artifacts. TL;DR: There are two remaining generic issues and probably > a small number of remaining port-specific differences. > > 1. Building install ISOs with `makefs` doesn't result in identical > filesystem images, as the ISOs contain a large number of files > _not_ listed in the manifest. These files retain their host's > user/group identity. > https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58196 > > 2. Some ports use CTF debugging data, which is (due to n unknown > root cause) different when created with NetBSD compared to CTF > data generated on a Linux host. (I suspect that data is sorted > somewhere, but that sort isn't stable for elements considered > "equal".) > https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58197 > > A third problem was that my local tarballs of generated release > artifacts wasn't stable, too, but an update from GNU tar 1.34 -> 1.35 > seems to have solved this issue. > > > Here is a table of current build results (NetBSD current as of a few > days ago, plus a small number of patches that still need to be > reviewed, submitted & merged): > > mostly ok: > If checked [x], there are no differences within the release > files, except the ISO images. > If checked [!], there are known differences that need to be > checked. (May be CTF-related, though.) > CTF: > If checked [!], this arch (aarch64, i386, amd64, earm) uses > non-reproducible CTF debug infos. > If checked [-], this arch does not use CTF debug infos. > ISO: > If checked [!], the ISO file contain differences (due to > ISO9660 owner/group), but the actual content > files are okay. > If checked [-], this port does not build ISO images. > If checked [x], this ISO(s) are bit-identical! > perfect: > If checked [x], a tarball from result artifact created on a > Linux build host is identical to a tarball > created from NetBSD-built artifacts. > Not tested for many ports/archs as it was > pointless with GNU tar 1.34. > > mst ok CTF ISO perfect > acorn32-earmv4 [!] [!] [!] [ ] > algor-mips64el [x] [-] [-] [x] > algor-mipsel [!] [-] [ ] [ ] Minimal > differences in algor/binary/kernel/netbsd-P5064.gz / > algor/binary/sets/kern-P5064.tgz > alpha-alpha [x] [-] [!] [ ] > amd64-x86_64 [!] [!] [!] [ ] > amiga-m68k [x] [-] [!] [ ] > amigappc-powerpc [x] [-] [-] [ ] > arc-mips64el [ ] [ ] [ ] [ ] Fails to > build: mipsel--netbsd-gcc: not found: No such file or directory (calling > 32bit GCC while only 64bit is configured) > arc-mipsel [x] [-] [!] [ ] > atari-m68k [x] [-] [!] [ ] > bebox-powerpc [x] [-] [-] [ ] > cats-earmv4 [!] [!] [!] [ ] > cesfic-m68k [x] [-] [-] [ ] > cobalt-mips64el [x] [-] [!] [ ] > cobalt-mipsel [x] [-] [!] [ ] > dreamcast-sh3el [!] [-] [!] [ ] Minimal > differences in dreamcast/binary/kernel/netbsd-GENERIC_MD{.bin,}.gz (like > algor-mipsel) > emips-mipseb [x] [-] [!] [ ] > epoc32-earmv4 [!] [!] [!] [ ] > evbarm-aarch64 [!] [!] [-] [ ] > evbarm-aarch64eb [!] [!] [-] [ ] > evbarm-earmv4 [!] [!] [-] [ ] > evbarm-earmv4eb [!] [!] [-] [ ] > evbarm-earmv5 [ ] [ ] [ ] [ ] Fails to > build > evbarm-earmv5eb [ ] [ ] [ ] [ ] Fails to > build > evbarm-earmv5hf [ ] [ ] [ ] [ ] Fails to > build > evbarm-earmv5hfeb [ ] [ ] [ ] [ ] Fails to > build > evbarm-earmv6 [!] [!] [-] [ ] > evbarm-earmv6eb [!] [!] [-] [ ] > evbarm-earmv6hf [!] [!] [-] [ ] > evbarm-earmv6hfeb [!] [!] [-] [ ] > evbarm-earmv7 [!] [!] [-] [ ] > evbarm-earmv7eb [!] [!] [-] [ ] > evbarm-earmv7hf [!] [!] [-] [ ] > evbarm-earmv7hfeb [!] [!] [-] [ ] > evbcf-coldfire [ ] [ ] [ ] [ ] Fails to > build: checking target system type... Invalid configuration > `m5407--netbsdelf': machine `m5407-' not recognized > evbmips-mips64eb [x] [-] [!] [ ] > evbmips-mips64el [x] [-] [!] [ ] > evbmips-mipseb [!] [-] [-] [ ] > evbmips-mipsel [!] [-] [-] [ ] > evbmips-mipsn64eb [x] [-] [!] [ ] > evbmips-mipsn64el [x] [-] [!] [ ] > evbppc-powerpc [!] [-] [!] [ ] > evbppc-powerpc64 [x] [-] [!] [ ] > evbsh3-sh3eb [x] [-] [!] [ ] > evbsh3-sh3el [x] [-] [!] [ ] > ews4800mips-mipseb [x] [-] [!] [ ] > hp300-m68k [x] [-] [!] [ ] > hpcarm-earmv4 [!] [!] [-] [ ] > hpcmips-mipsel [x] [-] [!] [ ] > hpcsh-sh3el [x] [-] [!] [ ] > hppa-hppa [!] [-] [!] [ ] > i386-i386 [!] [!] [!] [ ] > ia64-ia64 [x] [-] [!] [ ] > ibmnws-powerpc [x] [-] [!] [ ] > iyonix-earm [!] [!] [-] [ ] > landisk-sh3el [x] [-] [-] [ ] > luna68k-m68k [x] [-] [!] [ ] > mac68k-m68k [x] [-] [!] [ ] > macppc-powerpc [x] [-] [!] [ ] > macppc-powerpc64 [x] [-] [!] [ ] > mipsco-mipseb [x] [-] [!] [ ] > mmeye-sh3eb [x] [-] [!] [ ] > mvme68k-m68k [x] [-] [!] [ ] > mvmeppc-powerpc [x] [-] [!] [ ] > netwinder-earmv4 [!] [!] [!] [ ] > news68k-m68k [x] [-] [!] [ ] > newsmips-mipseb [x] [-] [!] [ ] > next68k-m68k [x] [-] [!] [ ] > ofppc-powerpc [x] [-] [!] [ ] > ofppc-powerpc64 [ ] [ ] [ ] [ ] Failed to > build (fails to link 32 vs 64 bit code) > or1k-or1k [ ] [ ] [ ] [ ] Failed to > build (No rule to make target > `/var/lib/laminar/run/netbsd-or1k-or1k/31/NetBSD-src/tools/gcc/../../external/gpl3/gcc.old/dist/gcc/config/or1k/or1k.md') > playstation2-mipsel [ ] [ ] [ ] [ ] Failed to > build (/root/NetBSD-src/usr.sbin/crash/obj/mips/mutex.h:50:2: error: > unknown type name 'ipl_cookie_t') > pmax-mips64el [x] [-] [!] [ ] > pmax-mipsel [x] [-] [!] [ ] > prep-powerpc [x] [-] [!] [ ] > riscv-riscv32 [!] [-] [!] [ ] Diff in > [base]/lib/libc.so.12.221 (.eh_frame_hdr) > riscv-riscv64 [!] [-] [!] [ ] Diff in > base{,32}.tgz > rs6000-powerpc [x] [-] [-] [ ] > sandpoint-powerpc [x] [-] [!] [ ] > sbmips-mips64eb [x] [-] [-] [x] > sbmips-mips64el [x] [-] [-] [x] > sbmips-mipseb [x] [-] [-] [x] > sbmips-mipsel [!] [-] [-] [ ] Diff in > sbmips/binary/kernel/netbsd-GENERIC.gz, sbmips/binary/sets/kern-GENERIC.tgz > sgimips-mips64eb [ ] [ ] [ ] [ ] Failed to > build: nbmake[4]: don't know how to make netbsd-INSTALL64_IP2x.gz. Stop > sgimips-mipseb [x] [-] [!] [ ] > shark-earmv4 [!] [!] [-] [ ] > sparc-sparc [!] [-] [!] [ ] Diff in > sparc/installation/bootfs/instfs.tgz, rootfs.tgz > sparc64-sparc64 [x] [-] [!] [ ] > sun2-m68000 [!] [-] [!] [ ] Diff in > sun2/binary/kernel/netbsd-RAMDISK.gz > sun3-m68k [!] [-] [!] [ ] Diff in > sun3/binary/kernel/netbsd-RAMDISK3X.gz, netbsd-RAMDISK.gz > vax-vax [x] [-] [!] [ ] > virt68k-m68k [x] [x] [-] [x] > x68k-m68k [x] [-] [!] [ ] > zaurus-earm [!] [!] [!] [ ] Diff in > x11 (date in docs) > > Ignoring needed fixes for ISO and CTF generation, NetBSD is on a good > track reaching reproducible build results even on quite different > build operating systems! That's a nice outcome! > > Thanks, > Jan-Benedict > PS: Special thanks to Christos and other people who had a look at my > occasional GNATS PRs! > > -- >
