And now I have to rebase my Python 3.13 PR to see if 3.13 also compiles
well :D

On Fri, Jul 11, 2025 at 6:53 PM Aritra Basu <aritrabasu1...@gmail.com>
wrote:

> Yay! 🙌
> --
> Regards,
> Aritra Basu
>
> On Fri, 11 Jul 2025, 9:48 pm Jarek Potiuk, <ja...@potiuk.com> wrote:
>
> > Ok. We got it merged again !
> https://github.com/apache/airflow/pull/53150
> > is merged. Once you build breeze images after rebase, python there will
> be
> > built directly from Python official sources (latest released patchlevel
> for
> > each version - and we have automation to upgrade them soon after new
> > patchlevel are released).
> >
> > We are going to try it for a week or so and see if it all looks good, we
> > will proceed to apply it to PROD images.
> >
> > J.
> >
> >
> > On Sat, Jul 5, 2025 at 10:24 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I merget it too early - reverting here
> > > https://github.com/apache/airflow/pull/52900 as I found an issue with
> > > other python versions that I overlooked
> > >
> > > On Sat, Jul 5, 2025 at 10:06 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> FYI - we just merged Aritra's change to build Python in CI image from
> > >> sources https://github.com/apache/airflow/pull/52265 -> we will run
> it
> > >> for a while in CI, i also want to test some self-upgrade scenarios.
> > >>
> > >> Then we will look at applying it to the PROD image if we find no
> > >> surprises.
> > >>
> > >> It looks really good and we might have slightly more secure images
> > >> produced (i.e. getting read of some libraries CVEs faster than the
> > >> "official" images.
> > >>
> > >> Good job Aritra!
> > >> J.
> > >>
> > >>
> > >> On Thu, Jun 26, 2025 at 9:16 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >>> It looks like building from sources on our CI using base debian image
> > >>> (with enabled optimizations) "just works" and takes 1m36s. -> and
> > taking
> > >>> into account that we already have pretty sophisticated CI-level
> remote
> > >>> caching that will allow to rebuild it only when either base image is
> > >>> released or python is released - building sounds like a very good
> idea.
> > >>>
> > >>> J.
> > >>>
> > >>> On Thu, Jun 26, 2025 at 7:02 PM Damian Shaw <
> > >>> ds...@striketechnologies.com> wrote:
> > >>>
> > >>>> Going from source is certainly going to give the most control, I
> > >>>> personally choose python-build-standalone for my own images rather
> > than
> > >>>> building from source to outsource all the build choices, such as
> what
> > >>>> optimization flags to enable, what compiler tool chains to use, etc.
> > >>>>
> > >>>> Damian
> > >>>>
> > >>>>
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: Jarek Potiuk <ja...@potiuk.com>
> > >>>> Sent: Thursday, June 26, 2025 10:15 AM
> > >>>> To: dev@airflow.apache.org
> > >>>> Subject: Re: [DISCUSS] Switching base Python container images to
> > >>>> python-standalone ?
> > >>>>
> > >>>> Thanks Damian - very insightful. And yes - I did not really want to
> > >>>> "diminish" the value of community images and work of the
> maintainers,
> > it
> > >>>> was really more on the "we base our image security on the
> assumptions
> > that
> > >>>> it's coming from "official" sources and surely there are some
> > guardrails" -
> > >>>> which turned out to be not really true.
> > >>>>
> > >>>> But.... We might not have to even use python standalone.
> > >>>>
> > >>>> Aritra just tested installation of Python straight from sources ->
> > >>>> https://github.com/apache/airflow/pull/52265 and I was actually
> > pretty
> > >>>> surprised how non-problematic and fast it was - and it seems it
> > passes all
> > >>>> our CI tests.  We just need to add a little pre-commit magic to get
> > >>>> notified when we should update patchlevel version of Python when a
> > new one
> > >>>> is released, and we should be able to try it out in CI image - once
> > it gets
> > >>>> battle-tested with CI/breeze etc. we can transfer this to PROD image
> > as
> > >>>> well. We already have an idea how to do it - our PROD images are
> > optimized
> > >>>> for size and do not contain "build essentials" - but I think we
> > should be
> > >>>> able to build Python in the "build" segment and simply copy
> resulting
> > >>>> binaries to the "main" segment - since in both cases we use the same
> > base
> > >>>> image, such 1-1 copy should **just work** - we already do the same
> > with
> > >>>> installed python packages - we install them (including building when
> > >>>> needed) in build segment and we copy-over the installed .venv to the
> > >>>> main segment.
> > >>>>
> > >>>> So ... we might even not need a discussion - installing from Python
> > >>>> sources is THE BEST
> > >>>>
> > >>>> J.
> > >>>>
> > >>>>
> > >>>> On Wed, Jun 25, 2025 at 6:25 PM Damian Shaw <
> > >>>> ds...@striketechnologies.com>
> > >>>> wrote:
> > >>>>
> > >>>> > First, I would like to thank the community members who have been
> > >>>> > maintaining the Python docker images, it's one of those thankless
> > >>>> > opensource infrastructure volunteer roles that they've been doing
> > for
> > >>>> > a long time. Unfortunately Docker assigns the title "Official
> Image"
> > >>>> > for various community run images, which creates a misconception on
> > the
> > >>>> > guarantees being provided, and if I were a suspicious person I
> would
> > >>>> > say Docker creates this misconception on purpose to both get free
> > work
> > >>>> > from community members and make Docker seem more supported by
> third
> > >>>> > party organizations than it actually is.
> > >>>> >
> > >>>> > On the topic of python-build-standalone, I've been using it in
> > >>>> > production for several months now and I'm fairly happy with it.
> > >>>> >
> > >>>> > However, one minor reproducibility issue I have when installing
> > >>>> > python-build-standalone via uv, is that uv does not have an
> ability
> > to
> > >>>> > pin to a specific build between uv versions, as uv hard codes a
> > >>>> > mapping of Python version and platform to a specific build and
> then
> > >>>> > updates that mapping between releases. So, updating the version of
> > uv
> > >>>> > between runs, or having two users run different versions of uv to
> > >>>> > initialize the environment can change the results.
> > >>>> >
> > >>>> > While normally such build changes are trivial, if you look at the
> uv
> > >>>> > 0.7.8 and 0.7.9 changelogs you will see that sometimes they can
> have
> > >>>> > significant
> > >>>> > impact: https://github.com/astral-sh/uv/releases/tag/0.7.8. Also,
> > >>>> this
> > >>>> > email finally prompted make an issue on this topic:
> > >>>> > https://github.com/astral-sh/uv/issues/14263.
> > >>>> >
> > >>>> > There are other ways to source python-build-standalone, such as
> > >>>> > pbs-installer or writing your own script, but I've not yet spent
> any
> > >>>> > time investigating them, so I can't comment on them.
> > >>>> >
> > >>>> > Damian
> > >>>> >
> > >>>> > -----Original Message-----
> > >>>> > From: Jarek Potiuk <ja...@potiuk.com>
> > >>>> > Sent: Wednesday, June 25, 2025 10:44 AM
> > >>>> > To: dev@airflow.apache.org
> > >>>> > Subject: [DISCUSS] Switching base Python container images to
> > >>>> > python-standalone ?
> > >>>> >
> > >>>> > Hello here,
> > >>>> >
> > >>>> > Together with Aritra we are looking into adding a few more things
> to
> > >>>> > our images (golang tool chain for CI image), also Shahar is
> > >>>> > experimenting with Rust tool chain and I also recently realized
> (by
> > >>>> > some of the issues we had) that 'Docker Official Python Image'
> that
> > is
> > >>>> > part of 'Official Program' [1] is not as 'Official' as I thought
> so
> > we
> > >>>> > discuss about changing the base of our images (first CI and then
> > when
> > >>>> > we see it works fine - PROD)
> > >>>> >
> > >>>> > Currently we are using the 'Official' image - but after some
> issues
> > >>>> > and discussions with people at PyCon and FOSS Backstage (I had a
> > >>>> > chance to talk to Python maintainers and even had a few beers with
> > >>>> > them) - it turned out that the official Python Image' is
> maintained
> > by
> > >>>> > 'a community's which really is a few pretty random people - and
> that
> > >>>> > explains for example why we have sometimes unpacked security
> > >>>> > vulnerabilities in setuptools etc. - because they made some
> > >>>> > compatibility choices and decisions that do not allow them to
> > upgrade
> > >>>> > easily, also they had some delays in releasing updated Python
> > >>>> > versions. And Docker does not **really** do much vetting there.
> > >>>> >
> > >>>> > So I think it would be good to switch how we build the base for
> our
> > >>>> images.
> > >>>> > And following the experience of `uv python` [2] - it seems that
> > maybe
> > >>>> > using "python-standalone" [3] project is a good alternative. It's
> > >>>> > managed by Astral now (so yes - another dependency on them), but
> > what
> > >>>> > you have with it you have practically 100% complete Python
> > interpreter
> > >>>> > installed in seconds.
> > >>>> > We could continue using debian-slim as a "base, base image" - and
> > >>>> > install python using "python-standalone". There are a few
> > >>>> > incompatibilities [4] of the distributions of Python, but there
> are
> > >>>> > very few and mostly related to some obscure systems
> (compatibilities
> > >>>> > with terminal in REPL, and gtk / UI integration that is anyhow not
> > >>>> > really working in "standard" Python distributions).
> > >>>> >
> > >>>> > I would love to hear what you think - happy to get any feedback/
> > >>>> > insights, suggestions and answer additional questions, provide
> some
> > >>>> > links to past "troubles" we had with Python "Official" images etc.
> > >>>> >
> > >>>> > J.
> > >>>> >
> > >>>> >
> > >>>> > [1] Official Python Images - https://hub.docker.com/_/python [2]
> UV
> > >>>> > Python installation -
> > >>>> https://docs.astral.sh/uv/guides/install-python/
> > >>>> > [3] Python Standalone project -
> > >>>> > https://github.com/astral-sh/python-build-standalone
> > >>>> > [4] Python Standalone incompatibilities -
> > >>>> >
> > >>>>
> > https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html
> > >>>> > ________________________________
> > >>>> >  Strike Technologies, LLC (“Strike”) is part of the GTS family of
> > >>>> > companies. Strike is a technology solutions provider, and is not a
> > >>>> > broker or dealer and does not transact any securities related
> > business
> > >>>> > directly whatsoever. This communication is the property of Strike
> > and
> > >>>> > its affiliates, and does not constitute an offer to sell or the
> > >>>> > solicitation of an offer to buy any security in any jurisdiction.
> It
> > >>>> > is intended only for the person to whom it is addressed and may
> > >>>> > contain information that is privileged, confidential, or otherwise
> > >>>> protected from disclosure.
> > >>>> > Distribution or copying of this communication, or the information
> > >>>> > contained herein, by anyone other than the intended recipient is
> > >>>> > prohibited. If you have received this communication in error,
> please
> > >>>> > immediately notify Strike at i...@striketechnologies.com, and
> > delete
> > >>>> and destroy any copies hereof.
> > >>>> > ________________________________
> > >>>> >
> > >>>> > CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> > >>>> > attachments are intended solely for the addressee. This
> transmission
> > >>>> > is covered by the Electronic Communications Privacy Act, 18 U.S.C
> > >>>> > ''2510-2521. The information contained in this transmission is
> > >>>> > confidential in nature and protected from further use or
> disclosure
> > >>>> > under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be
> > >>>> > subject to attorney-client or other legal privilege. Your use or
> > >>>> > disclosure of this information for any purpose other than that
> > >>>> > intended by its transmittal is strictly prohibited, and may
> subject
> > >>>> > you to fines and/or penalties under federal and state law. If you
> > are
> > >>>> > not the intended recipient of this transmission, please DESTROY
> ALL
> > >>>> > COPIES RECEIVED and confirm destruction to the sender via return
> > >>>> transmittal.
> > >>>> >
> > >>>> ________________________________
> > >>>>  Strike Technologies, LLC (“Strike”) is part of the GTS family of
> > >>>> companies. Strike is a technology solutions provider, and is not a
> > broker
> > >>>> or dealer and does not transact any securities related business
> > directly
> > >>>> whatsoever. This communication is the property of Strike and its
> > >>>> affiliates, and does not constitute an offer to sell or the
> > solicitation of
> > >>>> an offer to buy any security in any jurisdiction. It is intended
> only
> > for
> > >>>> the person to whom it is addressed and may contain information that
> is
> > >>>> privileged, confidential, or otherwise protected from disclosure.
> > >>>> Distribution or copying of this communication, or the information
> > contained
> > >>>> herein, by anyone other than the intended recipient is prohibited.
> If
> > you
> > >>>> have received this communication in error, please immediately notify
> > Strike
> > >>>> at i...@striketechnologies.com, and delete and destroy any copies
> > >>>> hereof.
> > >>>> ________________________________
> > >>>>
> > >>>> CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any
> > >>>> attachments are intended solely for the addressee. This transmission
> > is
> > >>>> covered by the Electronic Communications Privacy Act, 18 U.S.C
> > ''2510-2521.
> > >>>> The information contained in this transmission is confidential in
> > nature
> > >>>> and protected from further use or disclosure under U.S. Pub. L.
> > 106-102,
> > >>>> 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or
> > other
> > >>>> legal privilege. Your use or disclosure of this information for any
> > purpose
> > >>>> other than that intended by its transmittal is strictly prohibited,
> > and may
> > >>>> subject you to fines and/or penalties under federal and state law.
> If
> > you
> > >>>> are not the intended recipient of this transmission, please DESTROY
> > ALL
> > >>>> COPIES RECEIVED and confirm destruction to the sender via return
> > >>>> transmittal.
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org
> > >>>>
> > >>>
> >
>

Reply via email to