Hello everyone, A follow-up based on what we went through in CI in the last few weeks, because I think it is a good preview of what our users will see when they extend the 3.4.0 images.
TL;DR; I am leaning towards releasing legacy images, as I anticipate a number of issues our users who want to extend the images might have and we will have to help them solve those. Today's issue: "Test examples of PROD image building" started failing without any change on our side [1]. The hardened base image we mirrored still had libbz2-1.0 1.0.8-6+dhi2, while the dhi.io apt repository had already moved on to a bzip2 that needs exactly libbz2-1.0 1.0.8-6+deb13u0+dhi2. So "apt-get install build-essential" (build-essential -> dpkg-dev -> bzip2) failed with "Unsatisfied dependencies ... held broken packages". Other users hit the same thing at the same time [2], and Docker fixed it only in the images of the latest Python patch releases. A similar thing happened earlier this month with perl [3]: dhi.io temporarily dropped perl from its repository, the images still shipped Docker's perl-base, so apt fell back to Debian's perl, which needs Debian's perl-base. Anything that installed git or build-essential failed. These are not one-off bugs. They come from how the hardened images are built: there are two sources of apt packages in the image, stock Debian and Docker's dhi.io repository, both moving, neither snapshotted. The packages preinstalled in the image are Docker's builds, and whenever the two repositories get out of sync, apt picks a package from one source that needs an exact version from the other, and the install fails. Docker treats these as incidents and fixes them one by one, but the pattern stays. For our CI, #76023 [4] (merged today) already resolves the "stable CI" vs. "upgrade to get security fixes" conundrum: - The base images are pinned to digests, and a new base image only reaches our builds through the "Upgrade important CI environment" PR, after the full CI has passed on it. - The mirror refreshes every six hours, and a moved tag triggers the upgrade check, so we pick up Docker's security fixes within hours rather than weeks. - apt is configured to prefer dhi.io packages over Debian's (without forcing downgrades), and this preference stays in the image, so images extending ours behave the same way. But that only protects our CI and the images we publish. Users extending our images run "apt-get install" against the live repositories whenever they build, so they will hit exactly the same class of errors, at random times, on Dockerfiles that worked yesterday. Many of them have never had to think about apt pinning or package origins, and the error messages are not easy to read. I think we will need to gather feedback from them after 3.4.0 and give examples and solutions in our docs for dealing with the differences in packages: how to read these errors, how to pin the base image by digest, and what to do when a package is missing from dhi.io or mismatched with Debian. This makes me go back to my original position: I now lean towards publishing the "-legacy" images for 3.4.x (option A), so users who hit this have something to fall back on while they migrate their extended images, instead of being blocked from upgrading Airflow. This is especially true because, as we saw today, this might happen pretty randomly - when someone extends the image from us - building the image **might** fail randomly as Docker Hardened APT repo will get newer, hardened versions of some dependencies. I would like to hear what others think, though. One more observation: we discover these issues early (and can fix them quickly) precisely because the CI image and the PROD image share the build logic, and our users extend the same Dockerfile. Our CI runs the same apt steps our users run, so it detects this kind of breakage before our users do, and the fixes we make for CI (like the apt preferences) go straight into the images they build on. I think sharing that code is a big advantage here. [1] https://github.com/apache/airflow/actions/runs/40968123434/job/123973461638 [2] https://github.com/docker-hardened-images/catalog/issues/689 [3] https://github.com/docker-hardened-images/catalog/issues/610 [4] https://github.com/apache/airflow/pull/76023 Jarek --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
