Note: Quotes below are from https://www.apache.org/legal/release-policy.html. This document also contains a wealth of information and answers to FAQs. I heartily recommend reading it thoroughly - it provides a lot of context and explains why certain decisions with releases have been made, why we need it as a "Foundation-wide" policy and it allows us to better understand the "spirit" of the policy, not only the "letter". Some people who do not look at legal and security implications of releases, supply chain issues etc. might think of the ASF release process as a "cargo cult" - but it's ALL BUT CARGO CULT. I've learned over the years that many of the decisions and parts of the policy that are MUST are absolutely necessary for a secure and "trustworthy" release process and many of the expectations of modern regulations (for example CRA - Cyber Resilience Act) are actually catching up with what some Open Source players (like the Apache Software Foundation) have "always" been doing.
More details: So am I right in saying in the source release we vote for: > 1. we include all source files (easy) > Yes. But also everything needed to run tests (we also extend that to building documentation in Airflow). Generally you should **only** need -source.tar.gz and do not need anything else (from the project - it's always the questions of "turtles all the way down" - i.e. you need OS, compilers, bash etc. - i.e. "appropriate platforms and tools". Ideally Free and Open Source (not always possible - for example if you build something for MacOS you have no choice to use whatever Apple gives you as compilers and signing tools). > Every ASF release MUST contain one or more source packages, which MUST be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools. A source release SHOULD not contain compiled code. > 2. we need to include the code to generate the convenience package(s) in > the source release. > Yes. You should not publish a convenience package (say for example .whl file in PyPI, or Docker container) if you are not able to build such a convenience package from your -sources.tar.gz (there should be instruction in that source.tar.gz how to do it - we keep them in INSTALL file at the top of the source-tar.gz package). Each convenience package should be clearly connected with a released -sources.tar.gz (usually by a version number that you release it with - so that the user can know which -source.tar.gz should be used to build the package on their own if they wish). Just a comment: of course we **know** that 9X% of the "consumption" of our releases go through .whl files from PyPI. We have no **doubts** about it - that very few people use the "-source.tar.gz". But there are people who do - there are for example a number of distributions that **only** use such published sources and never use .whl files nor even .sdist files. And there is a growing trend (connected with the raise of supply-chain attacks) and the "reproducible builds" initiative to promote and make better tooling and ecosystem for reproducible builds https://reproducible-builds.org/ - i.e. making sure that given "sources" you are able to reproduce bit-to-bit identical installable packages. And projects like OSS rebuild https://oss-rebuild.dev/ aim to automate the process of rebuilding and verification of reproducibility. I have **just** returned from Vienna from Reproducible Builds Summit where we worked together on a future of it: https://reproducible-builds.org/news/2025/08/20/reproducible-builds-summit-in-vienna/ - so we are working as an industry to make the practices we do in ASF more wide-spread and actually used to prevent supply chain issues. > > Clarification: do we need to have the build process automated as in > automated via github actions? Or is it enough to just include the code that > would be run to create the convenience packages? > No need to automate via GitHub Actions (actually even see below description on when CI automation can be used for releases - it's often not possible or desired) , at the very minimum you need to have documentation / scripts/ example commands on how you build convenience packages. In Airflow I take it as a pride point to run **all** our release tooling as part of our GitHub Actions but this is more something that protects release manager - because those tools have a tendency to break if not used for a while, so we make sure to use them daily (by a CI). We also have **some** of our convenience packages **actually** build with Github Actions - for example our Docker container **ARE** build in CI: with this (manually triggered) workflow https://github.com/apache/airflow/actions/workflows/release_dockerhub_image.yml . Our documentation (which you might also treat as a convenience package) is built and published using a set of Github Actions as well (triggered via local "breeze" release management tool -> https://github.com/apache/airflow/tree/main/docs. But for example PyPI and wheel packages are (for now) built locally by Release Manager (and PMC members verifying it for reproducibility) by the same "breeze" tool as part of our release process. Docs for building Airflow core release for example here ( https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md) - we have ~200 convenience packages released regularly :). We build them all in CI as well for every PR and in canary builds using the same 'breeze` tool - to make sure that they continue to work. Actually it's even required by the policy that the actual sources are signed (and in many cases it means that they are also built) on hardware controlled by the Release Manager. So CI automation (unless you meet certain requirements) should **not** be actually used. When Apache Trusted Releases will get out of Beta, we are planning to move **all** the build to GitHub Actions though and run the actual releases on the ASF managed infrastructure as described here: https://infra.apache.org/release-signing.html#automated-release-signing - there are certain pre-conditions necessary (like reproducible builds) in order to be able to release artifacts that were build on a hardware that is not controlled by the Release Manager. I hope it helps. J. > Cheers, > > Stefan > > On Sun, Oct 19, 2025 at 11:34 PM Jarek Potiuk <[email protected]> wrote: > > > *TL;DR; What you publish in PyPI are not "source packages" releasing of > > which is a legal act of the Foundation. They are convenience packages, so > > they do not have the same "restrictions" - however you should make sure > > that whatever you release as "convenience packages" also referred to as > > "compiled packages" that can be automatically built from the sources. See > > [1] * > > > > More details: > > > > You can see what we do in Airflow. We have different packages. Sources > are > > -sources.tar.gz, and there it's simply git-archive - which includes all > > typescript files, but none of the resulting "processed" and "minified" > > javascript. > > This is also why in airflow we have `hatch` as build backend - as there > we > > have a "custom' hook that: > > > > * installs node and all the tools and transpiles the javascript > > * also produces git hash when run from sources so that we have commit > hash > > info, not only version available in the UI when you look at version > > > > Then, we have separate whl and sdist packages. And when we release and > vote > > and do all the verifications by PMC members (including reproducibility > > checks), we do that for all of those: > > > > * sources.tar.gz > > * sdist.tar.gz > > * .whl > > > > Also this is fine to vote on because the code in .whl and .sdist has been > > really "automatically produced" (given the right tools and environment) > > from the sources, so as long as this condition is met, it's ok to release > > it. In fact the `.whl` and `sdist` packages are not the "source release" > > that you are supposed to vote on in the release process (only sources > are) > > - they are "convenience packages" that are result of processing those > > sources - even if pretty much all regular users would use those > convenience > > packages, this is the "sources.tar.gz" which is legally and officially > > released by the ASF and this is what legally three PMC members voting on > > make a "legal act of the Foundation". The convenience packages are just > ... > > convenience. > > > > We (in Airflow) do all the voting and releases and PMC checks of > checksums, > > signatures and reproducibility (our sdist and whl package are bit-to-bit > > reproducible) so that we know that the convenience packages were simply > > result of automated process applied to the sources - and 3 PMC members > > confirm it independently (this way we know that release manager did not > > inject anything - i.e. "trust no one" aka prevent "xz" kind of thing) - > but > > this is not technically required by ASF, this is something we do extra as > > "additional security measure". We do not do it (reproducibility check) > for > > container images (which are another type of convenience packages) > produced > > in DockerHub (but there we have automated action that builds such > > images and we use the PyPI packages released as convenience images. We > > decided to host those whl and sdist packages also in " > downloads.apache.org > > " > > (https://downloads.apache.org/airflow/3.1.0/ for example) alongside the > > -sources.tar.gz but technically we do not have to. In our case, also > > because of binary reproducibility it's really nice because even if you > are > > using PyPI to install those packages, you can binary compare those PyPI > > packages with those that are in "downloads" and they are **the same**. > > > > Again - the important part is the "automate" part. If you can take source > > packages released as -sources.tar.gz - and (using appropriate tools and > > environment) - produce the files that you package in convenience > packages, > > this is "OK" - automatically generated files are not even supposed to > have > > licence information. > > > > I hope it helps. > > > > [1] https://www.apache.org/legal/release-policy.html#compiled-packages > > > > J. > > > > > > On Mon, Oct 20, 2025 at 5:21 AM Stefan Krawczyk < > [email protected] > > > > > wrote: > > > > > Hey All, > > > > > > I think we're pretty much there to make a release: > > > > > > 1. Docs have ASF (download page will come once we release) > > > 2. Files have headers > > > 3. Removed dependency on mozilla licensed JS lib -- so all JS libs are > > > apache 2 compatible. > > > 4. Added requisite files: license, disclaimer. > > > > > > My only question is, what is considered "source" for burr since we > > package > > > JS into the python? > > > > > > Options: > > > 1. We don't package the JavaScript. This requires someone to npm > install > > > things. But IIUC this means that what we upload to pypi would be > > different. > > > i.e. it would contain "code" that wasn't voted on because we add in the > > JS > > > bundled code? > > > 2. We do package the JavaScript. The JavaScript would be minified -- so > > it > > > would be a transformation of the originally licensed code. This then > > means > > > what is in pypi matches, however we'd need to make sure that the > license > > > file contains mentions to the code that is not licensed with apache 2. > > > > > > I think option (2) should work. > > > > > > Any thoughts? Mentors? I'm happy to ask on the main incubator list > about > > > this too. > > > > > > Cheers, > > > > > > Stefan > > > > > >
