Wiki - https://fedoraproject.org/wiki/Changes/Package_builds_are_expected_to_be_reproducible Discussion thread - https://discussion.fedoraproject.org/t/f43-change-proposal-package-builds-are-expected-to-be-reproducible-sysyem-wide/147320
This is a proposed Change for Fedora Linux. This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee. == Summary == Over the last few releases, we changed our build infrastructure to make package builds reproducible. This is enough to reach 90%. The remaining issues need to be fixed in individual packages. After this Change, '''package builds are expected to be reproducible'''. Bugs will be filed against packages when an irreproducibility is detected. The goal is to have no fewer than 99% of package builds reproducible. A public service with package rebuild statistics and reports for individual packages is made available. (An instance of [https://github.com/kpcyrd/rebuilderd/ rebuilderd].) A script to make local rebuilds of historic koji builds is made available ([https://github.com/keszybz/fedora-repro-build fedora-repro-build]). == Owner == * ** Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]] ** Email: zbys...@in.waw.pl * ** Name: Davide Cavalca ** Email: dcava...@fedoraproject.org * ** Name: [[User:Jelly| Jelle van der Waa]] ** Email: jvand...@redhat.com == Detailed Description == We define "reproducible" as when a completely independent rebuild of a koji package produces rpms that are identical except for the build timestamps, signatures, and some associated metadata. The ''payload'' (i.e. packaged files) and important metadata are bit-for-bit identical. In 2023, [[Changes/ReproducibleBuildsClampMtimes]] added clamping of mtimes to `$SOURCE_DATE_EPOCH` and changed the process to produce `.pyc` files to take `$SOURCE_DATE_EPOCH` into account. In 2024, [[Changes/ReproduciblePackageBuilds]] introduced `add-determinism` into package builds. Along the way, various details in rpm itself and other tools were adjusted to increase reproducibility. With those changes, about 90% of package builds are reproducible. The initial "big" issues that affect all packages are now solved, and what remains are problems that require changes in individual packages. The next step is to '''ask maintainers to resolve reproducibility issues in their packages'''. The goal is to have 99% of packages reproducible in Fedora. To achieve this, bugzillas will be opened against packages when a rebuild reports differences. We are aware of some issues that cannot be fixed easily: * Haskell packages are not reproducible when compiled with more than one thread. Upstream is [https://gitlab.haskell.org/ghc/ghc/-/issues/12935 working on the issue], the next release of ghc may resolve the issue completely. * mingw packages have irreproducible debug data. * golang packages have irreproducible debug data ([https://pagure.io/fedora-reproducible-builds/project/issue/15 irreproducibility#15]). * the kernel uses an ephemeral key for module signatures. See [https://lore.kernel.org/lkml/20241225-module-hashes-v1-0-d710ce7a3...@weissschuh.net/],[https://lwn.net/Articles/1012946/] for a possible solution. * packages that are signed for SecureBoot use a private key (shim, grub2). * Some BuildRequires on srpms are architecture-dependent. This is mostly an artifact of how we prepare the environment for builds. It does not directly affect binary rpms. We will create tracker bugs for the issues that affect multiple packages (haskell, mingw, golang). We hope that those issues will be resolved either upstream or downstream. == Feedback == == Benefit to Fedora == (''The first three paragraphs are copied unchanged from [[Changes/ReproduciblePackageBuilds]].) Adding determinism (i.e., removing non-determinsim) enables the Fedora community to have confidence that, if given the same source code, build environment, build instructions, and metadata from the build artifacts, any party can recreate copies of the artifacts that are identical except for the signatures and some parts of metadata. Reproducibility of builds leads to packages of higher quality. It turns out that quite often those irreproducible bits are caused by an error or sloppiness in the code. In particular, any dependence on architecture in noarch packages is almost always unwanted and/or a bug. Test builds that check reproducibility will expose such instances. Reproducibility of builds makes it easier to develop packages: when a small change is made and a package is rebuilt (in the same environment), then with a reproducible package, the only difference is directly caused by the change. If the package is different every time it is rebuilt, making a comparison is much harder. Build reproducibility is a topic that is gaining in popularity. Major distributions like Debian, Arch, OpenSUSE, and NixOS are trying to achieve full reproducibility. By making b-r an expectation in Fedora, we avoid driving away people who consider b-r a requirement. We may even attract additional contributors who are interested in this topic, if we achieve better results than other distros. With 90% reproducibility we're on par, with 99% we can be the leader 😀. == Scope == * Proposal owners: ** Package [https://github.com/keszybz/fedora-repro-build fedora-repro-build] to allow local rebuilds of historical koji builds ** Make [https://github.com/kpcyrd/rebuilderd/ rebuilderd] work with Fedora packages and repos ** Stand up a public rebuilderd instance for Fedora rawhide ** Adjust `add-determinism` to handle new cases, if widespread issues are found and it's possible to handle them in the cleanup phase ** Open bugs against packages when irreproducibilities are detected, initially as a manual process ** Open a pull request for Packaging Guidelines * Other developers: ** Fix irreproducibility issues in individual packages ** Fix tooling issues that affect multiple packages, if possible ** Review the pull request for Packaging Guidelines ** Adjust various packager workflow descriptions in the wiki * Release engineering: [https://pagure.io/releng/issues #Releng issue number] * Policies and guidelines: ** Packaging Guidelines will be changed to say that packages **should** build reproducibly, and link to our docs ([https://docs.fedoraproject.org/en-US/reproducible-builds/ reproducible builds]) and upstream docs at [https://reproducible-builds.org/documentation reproducible-builds.org]. * Trademark approval: N/A (not needed for this Change) * Alignment with the Fedora Strategy: == Upgrade/compatibility impact == N/A == Early Testing (Optional) == We have the tooling in place to do a local rebuild of a koji build and compare the results. Packagers can do such rebuilds and verify the result or fix the issues if any are found. == How To Test == When the public rebuilderd instance is up, maintainers will be able to see reports and diffoscope output for their packages. They can then use this to learn about any issues in their packages. == User Experience == No user-visible changes. Users ''may'' do local rebuilds and expect that the results are reproducible. == Dependencies == == Contingency Plan == * Contingency mechanism: All the items in the Scope are additions to the current tooling and infrastructure, so there is no need for a contingency plan for them. If we back out of the whole idea, rather than just postponing it for example, Packaging Guidelines changes would probably need to be reverted and any bugzillas closed. * Contingency deadline: any time * Blocks release? no == Documentation == * [https://reproducible-builds.org/docs reproducible-builds.org/docs] * https://docs.fedoraproject.org/en-US/reproducible-builds/ == Release Notes == Reproducibility of package builds has improved. TBD% of package builds are now reproducible. -- Aoife Moloney Fedora Operations Architect Fedora Project Matrix: @amoloney:fedora.im IRC: amoloney -- _______________________________________________ devel-announce mailing list -- devel-annou...@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
