Hi Jarek,
cool! Thanks for all the efforts and the breath in preparing this! Added
some comments in PR gut greatly appreciate (1) that notes describe the
upper bindings and (2) that the reference image default stays with 3.12
as 3.13 lags critical providers being ready.
Jens
On 15.07.25 19:52, Jarek Potiuk wrote:
*TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews and
merge !!!! *
The PR is finally about to get green - after about 100 or so iterations (or
so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
making). https://github.com/apache/airflow/pull/46891. I would really
appreciate some final reviews and approvals.
Jens and Pavan had already made a number of comments over the loooong
course of this one and I **think** I addressed all of them. I am happy to
get more comments - maybe some of the choices I made are outdated or can be
improved - but now having the PR green it will be way easier to iterate on
it
It took a really long time and a number of iterations and some opinionated
choices. Mostly because FAB cannot be installed yet with Python 3.13 - but
it also gave me an opportunity to move out or run conditionally tests - for
all cases where FAB was still not completely removed from the core. One of
our tests (lowest-deps core Python 3.13) runs the complete suite of tests
for core where `fab-provider` and `flask-app-builder` are completely
missing from the CI environment (BTW. `uv sync` is an awesome feature of
`uv` and it made it possible). And a number of changes from other people
(Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to
be merged before we could get this one green).
So I have a kind request - for those who know the codebase well - please
take a look and let me know if you have any comments. Yes, it's a monstrous
PR and - unfortunately - it's not easily breakable into smaller ones, but I
tried to make it as easy to review as possible. I split it into 4
independent commits (which will still have to be squashed and merged
together):
1) Breeze changes for Python 3.13 - changes in breeze code that enable
Python 3.13
2) Pyproject.toml changes in Providers for Python 3.13 (including
dependencies and python_requires) - big number of files
3) README.rst changes for Providers for Python 3.13 (big number of files)
4) Actual changes that were necessary to get the PR green for Python 3.13
(smaller number, but more substantial changes in airflow-core and provider
code that makes Python 3.13 work)
Currently excluded providers for Python 3.13:
** apache.beam*
** apache.kafka*
** fab*
** **yandex*
** ydb*
They are excluded for various reasons - TODO:s left explaining the reasons
and we can gradually bring them in as blockers are removed). Detailed
summary of changes in the PR and copied below.
When you review the code, you can focus on the part that you are interested
in (1) - if you want to look at breeze and (4) if you want to look at code
changes, and skip rather easily absolutely necessary dependency requirement
changes in (2) and mostly-automated changes from (3).
Yep I am asking for a lot, sorry, but I can't see how I could break that
one down into separate PRs easily.
I have two kind requests to take a closer look specifically:
a) for *Pavan* -> if you could check the aws-wrangler removal replacing it
with botocore - for docs publishing. I am not 100% sure if I got it right,
I did test it but a second pair of hands would be great to test it
b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones
working on Waeviate in the past) -> I would really appreciate it if you
could test the Weaviate changes I made. Weaviate was one of the problematic
changes because dependency update for 3.13 "freed" upgrade to a newer
version of Waeviate client (>=4.10.0) that used a httpx rather than
requests - and we had some low-level mock tests for retry scenarios (429,
500, ConnectErrors). And in order to fix it, I had to not only have code in
tests for mocking either httpx or requests, but I also had to change the
retriable error detection in Weaviate provider. I **think** I made proper
changes there, but I have no experience/no Weaviate instance so I had not
tested it with the "real" weaviate - I updated the tests and mocked httpx
in the way I **think** it will work, but I am not 100% sure if I got it
right. Actual tests with weaviate would be great (and you had done such
tests before so it might be way easier for you).
J.
-------------
This is the summary of the changes made. Beware .... it's .....
LOOOONG.....
Add Python 3.13 support for Airflow.
Added Python 3.13 support across the codebase, including:
* CI/CD workflow updates to handle Python 3.13.
* Dockerfile and environment variable adjustments for compatibility.
* Documentation updates to list Python 3.13 as supported.
* Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
* Test and plugin manager adjustments for Python 3.13-specific
behaviors.
* Removal or skipping of tests and features not compatible with Python
3.13.
* Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
* Updated all provider README.rst files to:
* List Python 3.13 as a supported version.
* Adjust dependency requirements for Python 3.13 (e.g., pandas,
pyarrow, ray, etc.).
* Add or update notes about excluded or conditionally supported
dependencies for Python 3.13.
* Document any known incompatibilities or workarounds for specific
providers.
* Updated all provider pyproject.toml files and automation to:
* Add Python 3.13 to the supported classifiers.
* Adjust requires-python and dependency constraints for Python 3.13.
* Exclude Python 3.13 for providers that are not yet compatible (e.g.,
those depending on Flask AppBuilder).
* Add conditional dependencies for new versions of libraries required
by Python 3.13.
* Updated Breeze and related scripts to:
* Build and test with Python 3.13.
* Update documentation and release scripts to include Python 3.13.
* Ensure all dev tooling and test environments work with Python 3.13.
* Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
* some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
* k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when running tests in Python 3.13
* migration tests had to be excluded for Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
* Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
* Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
J.
On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
I also started to do more preparation and extracting things that we will
need to merge first and release providers with some limits lifted. We need
to make our providers released in PyPI prepared upfront because currently
we are not able to generate PyPI constraints for Python 3.13 - I already
have one of those merged - https://github.com/apache/airflow/pull/51994
where we have to exclude `ray` from Python 3.13 and release provider before
we get the PR "green" - similarly I have just created
https://github.com/apache/airflow/pull/52060 to all providers that are
currently limiting pandas (unnecessary it seems from my tests in 3.13 PR).
J.
On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <amoghdesai....@gmail.com>
wrote:
Thanks for taking this on Jarek.
These are not easy tasks to work on and I cannot think of a better person
to do it than you!
I will review it shortly this week.
I think it is a good thing to introduce in Airflow 3.1 and I would not be
too much in favour
to cherry pick it to the 3.x series, just for convenience sake.
Thanks & Regards,
Amogh Desai
On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
Hey here,
Over the last few days and weeks I made significant progress with Python
3.13 support and while the PR is not yet fully green - we are getting
close. I undrafted my PR recently:
https://github.com/apache/airflow/pull/46891
I have a kind request for others to start reviewing it, I will be happy
to
respond to questions and I am also happy to accept some fixups to solve
things in a better way if possible, I did a few hacks and left some
TODOs
that might need some creative ideas to be solved in a better, simpler
way.
I split the PR into several commits - and keep it continuously rebased
on
top of the latest main. I had to remove some providers (including FAB -
which is actually cool, because we can finally see that airflow
codebase is
finally working even without FAB being around) - some of them are yet to
implement Python 3.13 (for example Apache Beam is working on it) so we
might add them back soon.
Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
there were a number of things removed/changed and small behavioral
changes
that we relied on, also the dependencies are catching up way later, and
there are some tricky dependencies of ours that make things more complex
when it comes to selecting "min" versions of those. But we are finally
getting there.
Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
might also attempt to cherry-pick it for 3.0, but it might be too much
of a
hassle, so we can decide after we merge this one to main.
J.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
