In PLC4X we’re also working on enabling fully reproducible builds. As PLC4X contains Java, C, C#, Go, Python, … it was a bit of a challenge, but I think we’ll cross the finishing lines with our upcoming release.
We basically gave up on the idea of configuring our tooling to produce the same output on every Java Version, Operating-system and CPU architecture, we simply defined a reference platform in form of a Dockerfile and use docker-compose to “ship the reference build machine”. So the idea is to execute all release operations in this reference build machine and to also use it to validate releases. The building and staging of releases has already been finished, still need to finish some of the missing parts, but hopeful that we’ll reach the final goal soon. Chris Von: Gary Gregory <garydgreg...@gmail.com> Datum: Sonntag, 14. Januar 2024 um 20:28 An: security-disc...@community.apache.org <security-disc...@community.apache.org> Cc: builds <builds@apache.org> Betreff: Re: Reproducible builds [Airflow] -> done Congratulations Jarkek :-) I was glad to see a reference to https://maven.apache.org/guides/mini/guide-reproducible-builds.html I hope we can keep that Maven page up-to-date with whatever comes up so we only have to look in one place ; -) Gary On Sun, Jan 14, 2024, 2:06 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Hey everyone, > > I just wanted to share a little accomplishment I (mostly) implemented in > Airflow - I just merged the last PR to get fully reproducible builds for > all thePython artifacts we produce and publish in downloads.apache.org > (python whl, sdist packages, source tarballs). > > All our 90 or so artifacts are now fully reproducible and we check > reproducibility of them as a mandatory step of PMC verification when voting > the releases. Initially I thought it's not THAT needed for us in the Python > world, but I got the "let's be reproducible" bug implanted at the > "reproducible builds" talk at the ApacheCon in Halifax by Hervé Boutemy and > it stuck - until I got it completed. > > And yeah. It simplified quite a lot our artifact verification and made our > process much more robust. > > Arnout just created this page > https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds > where we might gather some notes and guidelines around reproducibility - > and I added some Python notes and links to our small snippets of making the > packages "nicer" reproducible. For example we have now pretty accurate - > yet reproducible - dates in the packages as we store source-date-epoch in > our repository and bump it automatically as part of our release preparation > process. Might be a good idea if others keep their notes there as well > to share experiences. > > As a side note (and maybe sparking a bit of a Java/Maven vs. Python > battle), I used to have a bit of an inferiority complex when comparing the > build system state in Python - where we did not have so good standards and > tooling was proliferated and complex. But as part of preparation there I > had to basically move Airflow to the modern packaging world of Python (we > accumulated quite some tech debt and had custom solutions there) but I > learned that with the approach of separate build frontend and build > backends, based on multiple PEP-standards, Python quite leapfrogged the > Java world IMHO. It's a bit of a marvel of what the Python Packaging team > accomplished in the last few years when it comes to standard adoption and > tooling. I am honestly quite impressed with it, and it's just the beginning > to be honest. And it makes our contributor's life so much easier - where > they can stick to whatever frontend they like (Hatch, poetry, flit, pip, > ....) and it seamlessly works with a nicely integrated and customizable > build backend of our choice (hatchling in this case). > > Looking forward to other stories there in the future. > > J. >
