Hi Ash, all, One more data point for this discussion - and it goes in the other direction: our Dockerfile is about to get *even more* complicated, to handle yet another case - the stability of the CI and PROD images we build and test during CI.
The DHI (Docker Hardened Images) base images move very quickly. Docker rebuilds them in place several times a day and updates the dhi.io apt repository together with them, dropping the package versions it replaced. Pinning the base image to a digest alone (already done) does not help because packages are still installed from that live repository. Consequently, the image built from the last known good SHA often fails to build once DHI publishes new dependencies. It happened 3 times in the last 2 weeks - most recently today, when every main image build broke on renumbered gcc/libgmp packages (fixed in https://github.com/apache/airflow/pull/76200). New PR: https://github.com/apache/airflow/pull/76225 adds a "frozen" OS-dependencies image: the base image with all system packages and database clients already installed. It is rebuilt regularly (daily, but only when its inputs change), and CI and release builds start from a digest-pinned version of it. That gives us stability - a PR builds the same image regardless of what dhi.io published an hour ago - while we still move to the latest base image every few days, whenever the "Upgrade important CI environment" PR succeeds with the new pins. That is our standard way to "eventually synchronize" with new, potentially breaking dependencies without breaking each PR. Ash - a suggestion: if you *really* want to get the vote and remove all the complexity of the various CI edge cases (like this one) from what our users see, one option would be to select only the cases we want to support, and implement automated synchronization (prek hook) from the Dockerfile we use in CI for testing (call it Dockerfile.PROD) to the "Dockerfile" we expose to our users. I thought about it a bit. If there is agreement on which use cases should be supported and which not - and I'd encourage you to lead that and call the vote - I could write automation that generates such a simpler user-facing Dockerfile, plus tests so that it is built and tested during our CI. That would be an interesting and useful thing to implement. I think if we want to test all the CI edge cases with the PROD image, and keep both—the seemingly contradicting stability of PRs AND quick catch-up with the latest security fixes in CI—the current PROD Dockerfile complexity will have to remain. However we can hide it from our users while keeping a single source of truth for the dependencies, installation stages and so on. The bare minimum I think should be supported is the multi-stage image, where build-essentials live in a "build" image, so users can get a small image without keeping an extra ~200 MB of compilers and other tools and libraries needed to build some Python packages. The Rust compiler is one that seems to be needed more and more often recently - some Python packages that embed Rust do not publish wheels for every Python version and architecture - especially some corporate packages that many of our current - and new - providers will need. So at the very least **that** functionality must stay - and we can likely simplify it if some of the edge cases - like installing packages from latest sources - is removed from the "user" Dockerfile. J. On Mon, Oct 26, 2026 at 7:35 PM Jarek Potiuk <[email protected]> wrote: > Develop a concrete proposal that tests all edge cases without impacting > our CI or adding duplication. I am happy to do the reviews. > > Also do not forget to call for at least LAZY CONSENSUS to remove features > our users got from us - and justify it. > > Though I feel when we remove features that some of our users might rely > to, VOTE is more appropriate (I would likely object LAZY CONSENSUS unless I > see a good justification of that). > > J. > > On Mon, Oct 26, 2026 at 6:46 PM Ash Berlin-Taylor <[email protected]> wrote: > >> >> >> > On 26 Oct 2026, at 14:26, Jarek Potiuk <[email protected]> wrote: >> > >> > This is something we agreed in: >> > >> > * AIP-26: >> > >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/132320277/AIP-26+Production-ready+Airflow+Docker+Image >> > (October 2019) >> >> Yup, hence why I’m making a proposal to change it now. > >
