Did anyone experience any suspicious problems with the breeze CI image prepared this way ?
I did not - python works as usual, all airflow features are working, I have not seen any "negative" side effect of changing python to source-built one - except slightly longer build time occasionally. If ther are no other observations or issues - Aritra, I tink it might be good time to look at trasplanting it to PROD image, Happy to either just help and review or do it and have you review it - up to you :) J. On Fri, Jul 11, 2025 at 6:54 PM Jarek Potiuk <ja...@potiuk.com> wrote: > And now I have to rebase my Python 3.13 PR to see if 3.13 also compiles > well :D > > On Fri, Jul 11, 2025 at 6:53 PM Aritra Basu <aritrabasu1...@gmail.com> > wrote: > >> Yay! 🙌 >> -- >> Regards, >> Aritra Basu >> >> On Fri, 11 Jul 2025, 9:48 pm Jarek Potiuk, <ja...@potiuk.com> wrote: >> >> > Ok. We got it merged again ! >> https://github.com/apache/airflow/pull/53150 >> > is merged. Once you build breeze images after rebase, python there will >> be >> > built directly from Python official sources (latest released patchlevel >> for >> > each version - and we have automation to upgrade them soon after new >> > patchlevel are released). >> > >> > We are going to try it for a week or so and see if it all looks good, we >> > will proceed to apply it to PROD images. >> > >> > J. >> > >> > >> > On Sat, Jul 5, 2025 at 10:24 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> > > I merget it too early - reverting here >> > > https://github.com/apache/airflow/pull/52900 as I found an issue with >> > > other python versions that I overlooked >> > > >> > > On Sat, Jul 5, 2025 at 10:06 AM Jarek Potiuk <ja...@potiuk.com> >> wrote: >> > > >> > >> FYI - we just merged Aritra's change to build Python in CI image from >> > >> sources https://github.com/apache/airflow/pull/52265 -> we will run >> it >> > >> for a while in CI, i also want to test some self-upgrade scenarios. >> > >> >> > >> Then we will look at applying it to the PROD image if we find no >> > >> surprises. >> > >> >> > >> It looks really good and we might have slightly more secure images >> > >> produced (i.e. getting read of some libraries CVEs faster than the >> > >> "official" images. >> > >> >> > >> Good job Aritra! >> > >> J. >> > >> >> > >> >> > >> On Thu, Jun 26, 2025 at 9:16 PM Jarek Potiuk <ja...@potiuk.com> >> wrote: >> > >> >> > >>> It looks like building from sources on our CI using base debian >> image >> > >>> (with enabled optimizations) "just works" and takes 1m36s. -> and >> > taking >> > >>> into account that we already have pretty sophisticated CI-level >> remote >> > >>> caching that will allow to rebuild it only when either base image is >> > >>> released or python is released - building sounds like a very good >> idea. >> > >>> >> > >>> J. >> > >>> >> > >>> On Thu, Jun 26, 2025 at 7:02 PM Damian Shaw < >> > >>> ds...@striketechnologies.com> wrote: >> > >>> >> > >>>> Going from source is certainly going to give the most control, I >> > >>>> personally choose python-build-standalone for my own images rather >> > than >> > >>>> building from source to outsource all the build choices, such as >> what >> > >>>> optimization flags to enable, what compiler tool chains to use, >> etc. >> > >>>> >> > >>>> Damian >> > >>>> >> > >>>> >> > >>>> >> > >>>> -----Original Message----- >> > >>>> From: Jarek Potiuk <ja...@potiuk.com> >> > >>>> Sent: Thursday, June 26, 2025 10:15 AM >> > >>>> To: dev@airflow.apache.org >> > >>>> Subject: Re: [DISCUSS] Switching base Python container images to >> > >>>> python-standalone ? >> > >>>> >> > >>>> Thanks Damian - very insightful. And yes - I did not really want to >> > >>>> "diminish" the value of community images and work of the >> maintainers, >> > it >> > >>>> was really more on the "we base our image security on the >> assumptions >> > that >> > >>>> it's coming from "official" sources and surely there are some >> > guardrails" - >> > >>>> which turned out to be not really true. >> > >>>> >> > >>>> But.... We might not have to even use python standalone. >> > >>>> >> > >>>> Aritra just tested installation of Python straight from sources -> >> > >>>> https://github.com/apache/airflow/pull/52265 and I was actually >> > pretty >> > >>>> surprised how non-problematic and fast it was - and it seems it >> > passes all >> > >>>> our CI tests. We just need to add a little pre-commit magic to get >> > >>>> notified when we should update patchlevel version of Python when a >> > new one >> > >>>> is released, and we should be able to try it out in CI image - once >> > it gets >> > >>>> battle-tested with CI/breeze etc. we can transfer this to PROD >> image >> > as >> > >>>> well. We already have an idea how to do it - our PROD images are >> > optimized >> > >>>> for size and do not contain "build essentials" - but I think we >> > should be >> > >>>> able to build Python in the "build" segment and simply copy >> resulting >> > >>>> binaries to the "main" segment - since in both cases we use the >> same >> > base >> > >>>> image, such 1-1 copy should **just work** - we already do the same >> > with >> > >>>> installed python packages - we install them (including building >> when >> > >>>> needed) in build segment and we copy-over the installed .venv to >> the >> > >>>> main segment. >> > >>>> >> > >>>> So ... we might even not need a discussion - installing from Python >> > >>>> sources is THE BEST >> > >>>> >> > >>>> J. >> > >>>> >> > >>>> >> > >>>> On Wed, Jun 25, 2025 at 6:25 PM Damian Shaw < >> > >>>> ds...@striketechnologies.com> >> > >>>> wrote: >> > >>>> >> > >>>> > First, I would like to thank the community members who have been >> > >>>> > maintaining the Python docker images, it's one of those thankless >> > >>>> > opensource infrastructure volunteer roles that they've been doing >> > for >> > >>>> > a long time. Unfortunately Docker assigns the title "Official >> Image" >> > >>>> > for various community run images, which creates a misconception >> on >> > the >> > >>>> > guarantees being provided, and if I were a suspicious person I >> would >> > >>>> > say Docker creates this misconception on purpose to both get free >> > work >> > >>>> > from community members and make Docker seem more supported by >> third >> > >>>> > party organizations than it actually is. >> > >>>> > >> > >>>> > On the topic of python-build-standalone, I've been using it in >> > >>>> > production for several months now and I'm fairly happy with it. >> > >>>> > >> > >>>> > However, one minor reproducibility issue I have when installing >> > >>>> > python-build-standalone via uv, is that uv does not have an >> ability >> > to >> > >>>> > pin to a specific build between uv versions, as uv hard codes a >> > >>>> > mapping of Python version and platform to a specific build and >> then >> > >>>> > updates that mapping between releases. So, updating the version >> of >> > uv >> > >>>> > between runs, or having two users run different versions of uv to >> > >>>> > initialize the environment can change the results. >> > >>>> > >> > >>>> > While normally such build changes are trivial, if you look at >> the uv >> > >>>> > 0.7.8 and 0.7.9 changelogs you will see that sometimes they can >> have >> > >>>> > significant >> > >>>> > impact: https://github.com/astral-sh/uv/releases/tag/0.7.8. >> Also, >> > >>>> this >> > >>>> > email finally prompted make an issue on this topic: >> > >>>> > https://github.com/astral-sh/uv/issues/14263. >> > >>>> > >> > >>>> > There are other ways to source python-build-standalone, such as >> > >>>> > pbs-installer or writing your own script, but I've not yet spent >> any >> > >>>> > time investigating them, so I can't comment on them. >> > >>>> > >> > >>>> > Damian >> > >>>> > >> > >>>> > -----Original Message----- >> > >>>> > From: Jarek Potiuk <ja...@potiuk.com> >> > >>>> > Sent: Wednesday, June 25, 2025 10:44 AM >> > >>>> > To: dev@airflow.apache.org >> > >>>> > Subject: [DISCUSS] Switching base Python container images to >> > >>>> > python-standalone ? >> > >>>> > >> > >>>> > Hello here, >> > >>>> > >> > >>>> > Together with Aritra we are looking into adding a few more >> things to >> > >>>> > our images (golang tool chain for CI image), also Shahar is >> > >>>> > experimenting with Rust tool chain and I also recently realized >> (by >> > >>>> > some of the issues we had) that 'Docker Official Python Image' >> that >> > is >> > >>>> > part of 'Official Program' [1] is not as 'Official' as I thought >> so >> > we >> > >>>> > discuss about changing the base of our images (first CI and then >> > when >> > >>>> > we see it works fine - PROD) >> > >>>> > >> > >>>> > Currently we are using the 'Official' image - but after some >> issues >> > >>>> > and discussions with people at PyCon and FOSS Backstage (I had a >> > >>>> > chance to talk to Python maintainers and even had a few beers >> with >> > >>>> > them) - it turned out that the official Python Image' is >> maintained >> > by >> > >>>> > 'a community's which really is a few pretty random people - and >> that >> > >>>> > explains for example why we have sometimes unpacked security >> > >>>> > vulnerabilities in setuptools etc. - because they made some >> > >>>> > compatibility choices and decisions that do not allow them to >> > upgrade >> > >>>> > easily, also they had some delays in releasing updated Python >> > >>>> > versions. And Docker does not **really** do much vetting there. >> > >>>> > >> > >>>> > So I think it would be good to switch how we build the base for >> our >> > >>>> images. >> > >>>> > And following the experience of `uv python` [2] - it seems that >> > maybe >> > >>>> > using "python-standalone" [3] project is a good alternative. It's >> > >>>> > managed by Astral now (so yes - another dependency on them), but >> > what >> > >>>> > you have with it you have practically 100% complete Python >> > interpreter >> > >>>> > installed in seconds. >> > >>>> > We could continue using debian-slim as a "base, base image" - and >> > >>>> > install python using "python-standalone". There are a few >> > >>>> > incompatibilities [4] of the distributions of Python, but there >> are >> > >>>> > very few and mostly related to some obscure systems >> (compatibilities >> > >>>> > with terminal in REPL, and gtk / UI integration that is anyhow >> not >> > >>>> > really working in "standard" Python distributions). >> > >>>> > >> > >>>> > I would love to hear what you think - happy to get any feedback/ >> > >>>> > insights, suggestions and answer additional questions, provide >> some >> > >>>> > links to past "troubles" we had with Python "Official" images >> etc. >> > >>>> > >> > >>>> > J. >> > >>>> > >> > >>>> > >> > >>>> > [1] Official Python Images - https://hub.docker.com/_/python >> [2] UV >> > >>>> > Python installation - >> > >>>> https://docs.astral.sh/uv/guides/install-python/ >> > >>>> > [3] Python Standalone project - >> > >>>> > https://github.com/astral-sh/python-build-standalone >> > >>>> > [4] Python Standalone incompatibilities - >> > >>>> > >> > >>>> >> > https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html >> > >>>> > ________________________________ >> > >>>> > Strike Technologies, LLC (“Strike”) is part of the GTS family of >> > >>>> > companies. Strike is a technology solutions provider, and is not >> a >> > >>>> > broker or dealer and does not transact any securities related >> > business >> > >>>> > directly whatsoever. This communication is the property of Strike >> > and >> > >>>> > its affiliates, and does not constitute an offer to sell or the >> > >>>> > solicitation of an offer to buy any security in any >> jurisdiction. It >> > >>>> > is intended only for the person to whom it is addressed and may >> > >>>> > contain information that is privileged, confidential, or >> otherwise >> > >>>> protected from disclosure. >> > >>>> > Distribution or copying of this communication, or the information >> > >>>> > contained herein, by anyone other than the intended recipient is >> > >>>> > prohibited. If you have received this communication in error, >> please >> > >>>> > immediately notify Strike at i...@striketechnologies.com, and >> > delete >> > >>>> and destroy any copies hereof. >> > >>>> > ________________________________ >> > >>>> > >> > >>>> > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any >> > >>>> > attachments are intended solely for the addressee. This >> transmission >> > >>>> > is covered by the Electronic Communications Privacy Act, 18 U.S.C >> > >>>> > ''2510-2521. The information contained in this transmission is >> > >>>> > confidential in nature and protected from further use or >> disclosure >> > >>>> > under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may >> be >> > >>>> > subject to attorney-client or other legal privilege. Your use or >> > >>>> > disclosure of this information for any purpose other than that >> > >>>> > intended by its transmittal is strictly prohibited, and may >> subject >> > >>>> > you to fines and/or penalties under federal and state law. If you >> > are >> > >>>> > not the intended recipient of this transmission, please DESTROY >> ALL >> > >>>> > COPIES RECEIVED and confirm destruction to the sender via return >> > >>>> transmittal. >> > >>>> > >> > >>>> ________________________________ >> > >>>> Strike Technologies, LLC (“Strike”) is part of the GTS family of >> > >>>> companies. Strike is a technology solutions provider, and is not a >> > broker >> > >>>> or dealer and does not transact any securities related business >> > directly >> > >>>> whatsoever. This communication is the property of Strike and its >> > >>>> affiliates, and does not constitute an offer to sell or the >> > solicitation of >> > >>>> an offer to buy any security in any jurisdiction. It is intended >> only >> > for >> > >>>> the person to whom it is addressed and may contain information >> that is >> > >>>> privileged, confidential, or otherwise protected from disclosure. >> > >>>> Distribution or copying of this communication, or the information >> > contained >> > >>>> herein, by anyone other than the intended recipient is prohibited. >> If >> > you >> > >>>> have received this communication in error, please immediately >> notify >> > Strike >> > >>>> at i...@striketechnologies.com, and delete and destroy any copies >> > >>>> hereof. >> > >>>> ________________________________ >> > >>>> >> > >>>> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any >> > >>>> attachments are intended solely for the addressee. This >> transmission >> > is >> > >>>> covered by the Electronic Communications Privacy Act, 18 U.S.C >> > ''2510-2521. >> > >>>> The information contained in this transmission is confidential in >> > nature >> > >>>> and protected from further use or disclosure under U.S. Pub. L. >> > 106-102, >> > >>>> 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client >> or >> > other >> > >>>> legal privilege. Your use or disclosure of this information for any >> > purpose >> > >>>> other than that intended by its transmittal is strictly prohibited, >> > and may >> > >>>> subject you to fines and/or penalties under federal and state law. >> If >> > you >> > >>>> are not the intended recipient of this transmission, please DESTROY >> > ALL >> > >>>> COPIES RECEIVED and confirm destruction to the sender via return >> > >>>> transmittal. >> > >>>> >> > >>>> >> --------------------------------------------------------------------- >> > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org >> > >>>> >> > >>> >> > >> >
