Hi Jarek,

That was quite a long PR to review. I focused most of my review time on the
areas
I’m more familiar with — (1) and (4). But I ended up going through the rest
as well 🙂

Great job getting that PR green! I really appreciate the inline comments
with active TODOS,
they are helpful and make it easier to follow the context.

Thanks & Regards,
Amogh Desai


On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler <j_scheff...@gmx.de.invalid>
wrote:

> Hi Jarek,
>
> cool! Thanks for all the efforts and the breath in preparing this! Added
> some comments in PR gut greatly appreciate (1) that notes describe the
> upper bindings and (2) that the reference image default stays with 3.12
> as 3.13 lags critical providers being ready.
>
> Jens
>
> On 15.07.25 19:52, Jarek Potiuk wrote:
> > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews
> and
> > merge !!!! *
> >
> > The PR is finally about to get green - after about 100 or so iterations
> (or
> > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
> > making). https://github.com/apache/airflow/pull/46891. I would really
> > appreciate some final reviews and approvals.
> >
> > Jens and Pavan had already made a number of comments over the loooong
> > course of this one and I **think** I addressed all of them. I am happy to
> > get more comments - maybe some of the choices I made are outdated or can
> be
> > improved - but now having the PR green it will be way easier to iterate
> on
> > it
> >
> > It took a really long time and a number of iterations and some
> opinionated
> > choices. Mostly because FAB cannot be installed yet with Python 3.13  -
> but
> > it also gave me an opportunity to move out or run conditionally tests -
> for
> > all cases where FAB was still not completely removed from the core. One
> of
> > our tests (lowest-deps core Python 3.13) runs the complete suite of tests
> > for core where `fab-provider` and `flask-app-builder` are completely
> > missing from the CI environment (BTW. `uv sync` is an awesome feature of
> > `uv` and it made it possible). And a number of changes from other people
> > (Jens,  Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to
> > be merged before we could get this one green).
> >
> > So I have a kind request - for those who know the codebase well - please
> > take a look and let me know if you have any comments. Yes, it's a
> monstrous
> > PR and - unfortunately - it's not easily breakable into smaller ones,
> but I
> > tried to make it as easy to review as possible. I split it into 4
> > independent commits (which will still have to be squashed and merged
> > together):
> >
> > 1) Breeze changes for Python 3.13 - changes in breeze code that enable
> > Python 3.13
> > 2) Pyproject.toml changes in Providers for Python 3.13 (including
> > dependencies and python_requires) - big number of files
> > 3) README.rst changes for Providers for Python 3.13 (big number of files)
> > 4) Actual changes that were necessary to get the PR green for Python 3.13
> > (smaller number, but more substantial changes in airflow-core and
> provider
> > code that makes Python 3.13 work)
> >
> > Currently excluded providers for Python 3.13:
> > ** apache.beam*
> > ** apache.kafka*
> > ** fab*
> > ** **yandex*
> > ** ydb*
> >
> > They are excluded for various reasons - TODO:s left explaining the
> reasons
> > and we can gradually bring them in as blockers are removed). Detailed
> > summary of changes in the PR and copied below.
> >
> > When you review the code, you can focus on the part that you are
> interested
> > in (1) - if you want to look at breeze and (4) if you want to look at
> code
> > changes, and skip rather easily absolutely necessary dependency
> requirement
> > changes in (2) and mostly-automated changes from (3).
> >
> > Yep I am asking for a lot, sorry, but I can't see how I could break that
> > one down into separate PRs easily.
> >
> > I have two kind requests to take a closer look specifically:
> >
> > a) for *Pavan* -> if you could check the aws-wrangler removal replacing
> it
> > with botocore - for docs publishing. I am not 100% sure if I got it
> right,
> > I did test it but a second pair of hands would be great to test it
> >
> > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones
> > working on Waeviate in the past) -> I would really appreciate it if you
> > could test the Weaviate changes I made. Weaviate was one of the
> problematic
> > changes because dependency update for 3.13 "freed" upgrade to a newer
> > version of Waeviate client (>=4.10.0) that used a httpx rather than
> > requests - and we had some low-level mock tests for retry scenarios (429,
> > 500, ConnectErrors). And in order to fix it, I had to not only have code
> in
> > tests for mocking either httpx or requests, but I also had to change the
> > retriable error detection in Weaviate provider. I **think** I made proper
> > changes there, but I have no experience/no Weaviate instance so I had not
> > tested it with the "real" weaviate - I updated the tests and mocked httpx
> > in the way I **think** it will work, but I am not 100% sure if I got it
> > right. Actual tests with weaviate would be great (and you had done such
> > tests before so it might be way easier for you).
> >
> > J.
> >
> > -------------
> >
> > This is the summary of the changes made. Beware  .... it's .....
> > LOOOONG.....
> >
> >      Add Python 3.13 support for Airflow.
> >
> >      Added Python 3.13 support across the codebase, including:
> >      * CI/CD workflow updates to handle Python 3.13.
> >      * Dockerfile and environment variable adjustments for compatibility.
> >      * Documentation updates to list Python 3.13 as supported.
> >      * Dependency and code changes for compatibility with new/changed
> >        libraries (e.g., numpy 2.x, greenlet, pendulum).
> >      * Test and plugin manager adjustments for Python 3.13-specific
> > behaviors.
> >      * Removal or skipping of tests and features not compatible with
> Python
> > 3.13.
> >      * Improved error handling and logging for unsupported
> plugins/providers
> >        on Python 3.13.
> >      * Updated all provider README.rst files to:
> >        * List Python 3.13 as a supported version.
> >        * Adjust dependency requirements for Python 3.13 (e.g., pandas,
> >          pyarrow, ray, etc.).
> >        * Add or update notes about excluded or conditionally supported
> >          dependencies for Python 3.13.
> >        * Document any known incompatibilities or workarounds for specific
> >          providers.
> >      * Updated all provider pyproject.toml files and automation to:
> >        * Add Python 3.13 to the supported classifiers.
> >        * Adjust requires-python and dependency constraints for Python
> 3.13.
> >        * Exclude Python 3.13 for providers that are not yet compatible
> (e.g.,
> >          those depending on Flask AppBuilder).
> >        * Add conditional dependencies for new versions of libraries
> required
> >          by Python 3.13.
> >      * Updated Breeze and related scripts to:
> >       * Build and test with Python 3.13.
> >       * Update documentation and release scripts to include Python 3.13.
> >       * Ensure all dev tooling and test environments work with Python
> 3.13.
> >      * Removed remaining FAB related code from core and removed tests for
> >        older FAB provider versions (<1.3.0)
> >      * some of the tests in core still depend on FAB - but we make them
> >        optional - both for parsing (we need to skip some imports) and
> >        execution - because they test a legacy behaviour of embedding
> >        Flask plugins that is still supported in core.
> >      * k8s tests now run also with SimpleAuthManager and
> SimpleAuthManager
> >        is used automatically when running tests in Python 3.13
> >      * migration tests had to be excluded for Python 3.13 because we
> cannot
> >        migrate down on Python 3.13 without FAB tables created. The tests
> >        will be brought back when FAB is supported or when we release 3.1
> >        and start working on 3.2, we should be able to migrate down to 3.1
> >      * Weaviate tests have been updated to handle case where
> weaviate-client
> >        is > 2.10.0 - because it started to use httpx instead of requests,
> >        and Python 3.13 actually unblocks using older version of weaviate
> >        (it was previously blocked indirectly via Flask dependencies and
> >        grpcio).
> >      * Added doc change in our rules on what is the "default" Python
> version
> >        for airflow images - handling the case where we cannot use
> "newest"
> >        Python version as default where not all "regular" image providers
> are
> >        supported in the latest version
> >
> > J.
> >
> > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> I also started to do more preparation and extracting things that we will
> >> need to merge first and release providers with some limits lifted. We
> need
> >> to make our providers released in PyPI prepared upfront because
> currently
> >> we are not able to generate PyPI constraints for Python 3.13 - I already
> >> have one of those merged - https://github.com/apache/airflow/pull/51994
> >> where we have to exclude `ray` from Python 3.13 and release provider
> before
> >> we get the PR "green" - similarly I have just created
> >> https://github.com/apache/airflow/pull/52060 to all providers that are
> >> currently limiting pandas (unnecessary it seems from my tests in 3.13
> PR).
> >>
> >> J.
> >>
> >>
> >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <amoghdesai....@gmail.com>
> >> wrote:
> >>
> >>> Thanks for taking this on Jarek.
> >>>
> >>> These are not easy tasks to work on and I cannot think of a better
> person
> >>> to do it than you!
> >>>
> >>> I will review it shortly this week.
> >>>
> >>> I think it is a good thing to introduce in Airflow 3.1 and I would not
> be
> >>> too much in favour
> >>> to cherry pick it to the 3.x series, just for convenience sake.
> >>>
> >>> Thanks & Regards,
> >>> Amogh Desai
> >>>
> >>>
> >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>> Hey here,
> >>>>
> >>>> Over the last few days and weeks I made significant progress with
> Python
> >>>> 3.13 support and while the PR is not yet fully green - we are getting
> >>>> close. I undrafted my PR recently:
> >>>>
> >>>> https://github.com/apache/airflow/pull/46891
> >>>>
> >>>> I have a kind request for others to start reviewing it, I will be
> happy
> >>> to
> >>>> respond to questions and I am also happy to accept some fixups to
> solve
> >>>> things in a better way if possible, I did a few hacks and left some
> >>> TODOs
> >>>> that might need some creative ideas to be solved in a better, simpler
> >>> way.
> >>>> I split the PR into several commits - and keep it continuously rebased
> >>> on
> >>>> top of the latest main. I had to remove some providers (including FAB
> -
> >>>> which is actually cool, because we can finally see that airflow
> >>> codebase is
> >>>> finally working even without FAB being around) - some of them are yet
> to
> >>>> implement Python 3.13 (for example Apache Beam is working on it) so we
> >>>> might add them back soon.
> >>>>
> >>>> Note - Python 3.13 was a WAY more difficult migration than Python
> 3.12 -
> >>>> there were a number of things removed/changed and small behavioral
> >>> changes
> >>>> that we relied on, also the dependencies are catching up way later,
> and
> >>>> there are some tricky dependencies of ours that make things more
> complex
> >>>> when it comes to selecting "min" versions of those. But we are finally
> >>>> getting there.
> >>>>
> >>>> Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
> >>>> might also attempt to cherry-pick it for 3.0, but it might be too much
> >>> of a
> >>>> hassle, so we can decide after we merge this one to main.
> >>>>
> >>>> J.
> >>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>
>

Reply via email to