BTW. The nice thing after those last changes is that MySQL, Postgres, MSSQL client libraries (MSSQL is there for msslq provider) - are installed from DHI, They are hardened as well - so we can be sure those clients contain latest security bugfixes very quickly.
On Sun, Nov 1, 2026 at 2:36 PM Jarek Potiuk <[email protected]> wrote: > Hi Ash, all, > > One more data point for this discussion - and it goes in the other > direction: our Dockerfile is about to get *even more* complicated, to > handle yet another case - the stability of the CI and PROD images we build > and test during CI. > > The DHI (Docker Hardened Images) base images move very quickly. Docker > rebuilds them in place several times a day and updates the dhi.io apt > repository together with them, dropping the package versions it replaced. > Pinning the base image to a digest alone (already done) does not help > because packages are still installed from that live repository. > Consequently, the image built from the last known good SHA often fails to > build once DHI publishes new dependencies. It happened 3 times in the last > 2 weeks - most recently today, when every main image build broke on > renumbered gcc/libgmp packages (fixed in > https://github.com/apache/airflow/pull/76200). > > New PR: https://github.com/apache/airflow/pull/76225 adds a "frozen" > OS-dependencies image: the base image with all system packages and database > clients already installed. It is rebuilt regularly (daily, but only when > its inputs change), and CI and release builds start from a digest-pinned > version of it. That gives us stability - a PR builds the same image > regardless of what dhi.io published an hour ago - while we still move to > the latest base image every few days, whenever the "Upgrade important CI > environment" PR succeeds with the new pins. That is our standard way to > "eventually synchronize" with new, potentially breaking dependencies > without breaking each PR. > > Ash - a suggestion: if you *really* want to get the vote and remove all > the complexity of the various CI edge cases (like this one) from what our > users see, one option would be to select only the cases we want to support, > and implement automated synchronization (prek hook) from the Dockerfile we > use in CI for testing (call it Dockerfile.PROD) to the "Dockerfile" we > expose to our users. > > I thought about it a bit. If there is agreement on which use cases should > be supported and which not - and I'd encourage you to lead that and call > the vote - I could write automation that generates such a simpler > user-facing Dockerfile, plus tests so that it is built and tested during > our CI. That would be an interesting and useful thing to implement. I think > if we want to test all the CI edge cases with the PROD image, and keep > both—the seemingly contradicting stability of PRs AND quick catch-up with > the latest security fixes in CI—the current PROD Dockerfile complexity will > have to remain. However we can hide it from our users while keeping a > single source of truth for the dependencies, installation stages and so on. > > The bare minimum I think should be supported is the multi-stage image, > where build-essentials live in a "build" image, so users can get a small > image without keeping an extra ~200 MB of compilers and other tools and > libraries needed to build some Python packages. The Rust compiler is one > that seems to be needed more and more often recently - some Python packages > that embed Rust do not publish wheels for every Python version and > architecture - especially some corporate packages that many of our current > - and new - providers will need. So at the very least **that** > functionality must stay - and we can likely simplify it if some of the edge > cases - like installing packages from latest sources - is removed from the > "user" Dockerfile. > > J. > > On Mon, Oct 26, 2026 at 7:35 PM Jarek Potiuk <[email protected]> wrote: > >> Develop a concrete proposal that tests all edge cases without impacting >> our CI or adding duplication. I am happy to do the reviews. >> >> Also do not forget to call for at least LAZY CONSENSUS to remove features >> our users got from us - and justify it. >> >> Though I feel when we remove features that some of our users might rely >> to, VOTE is more appropriate (I would likely object LAZY CONSENSUS unless I >> see a good justification of that). >> >> J. >> >> On Mon, Oct 26, 2026 at 6:46 PM Ash Berlin-Taylor <[email protected]> wrote: >> >>> >>> >>> > On 26 Oct 2026, at 14:26, Jarek Potiuk <[email protected]> wrote: >>> > >>> > This is something we agreed in: >>> > >>> > * AIP-26: >>> > >>> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/132320277/AIP-26+Production-ready+Airflow+Docker+Image >>> > (October 2019) >>> >>> Yup, hence why I’m making a proposal to change it now. >> >>
