*TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews and merge !!!! *
The PR is finally about to get green - after about 100 or so iterations (or so it seems - I opened it on Feb 19th (!) - so it's 5 months in the making). https://github.com/apache/airflow/pull/46891. I would really appreciate some final reviews and approvals. Jens and Pavan had already made a number of comments over the loooong course of this one and I **think** I addressed all of them. I am happy to get more comments - maybe some of the choices I made are outdated or can be improved - but now having the PR green it will be way easier to iterate on it It took a really long time and a number of iterations and some opinionated choices. Mostly because FAB cannot be installed yet with Python 3.13 - but it also gave me an opportunity to move out or run conditionally tests - for all cases where FAB was still not completely removed from the core. One of our tests (lowest-deps core Python 3.13) runs the complete suite of tests for core where `fab-provider` and `flask-app-builder` are completely missing from the CI environment (BTW. `uv sync` is an awesome feature of `uv` and it made it possible). And a number of changes from other people (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to be merged before we could get this one green). So I have a kind request - for those who know the codebase well - please take a look and let me know if you have any comments. Yes, it's a monstrous PR and - unfortunately - it's not easily breakable into smaller ones, but I tried to make it as easy to review as possible. I split it into 4 independent commits (which will still have to be squashed and merged together): 1) Breeze changes for Python 3.13 - changes in breeze code that enable Python 3.13 2) Pyproject.toml changes in Providers for Python 3.13 (including dependencies and python_requires) - big number of files 3) README.rst changes for Providers for Python 3.13 (big number of files) 4) Actual changes that were necessary to get the PR green for Python 3.13 (smaller number, but more substantial changes in airflow-core and provider code that makes Python 3.13 work) Currently excluded providers for Python 3.13: ** apache.beam* ** apache.kafka* ** fab* ** **yandex* ** ydb* They are excluded for various reasons - TODO:s left explaining the reasons and we can gradually bring them in as blockers are removed). Detailed summary of changes in the PR and copied below. When you review the code, you can focus on the part that you are interested in (1) - if you want to look at breeze and (4) if you want to look at code changes, and skip rather easily absolutely necessary dependency requirement changes in (2) and mostly-automated changes from (3). Yep I am asking for a lot, sorry, but I can't see how I could break that one down into separate PRs easily. I have two kind requests to take a closer look specifically: a) for *Pavan* -> if you could check the aws-wrangler removal replacing it with botocore - for docs publishing. I am not 100% sure if I got it right, I did test it but a second pair of hands would be great to test it b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones working on Waeviate in the past) -> I would really appreciate it if you could test the Weaviate changes I made. Weaviate was one of the problematic changes because dependency update for 3.13 "freed" upgrade to a newer version of Waeviate client (>=4.10.0) that used a httpx rather than requests - and we had some low-level mock tests for retry scenarios (429, 500, ConnectErrors). And in order to fix it, I had to not only have code in tests for mocking either httpx or requests, but I also had to change the retriable error detection in Weaviate provider. I **think** I made proper changes there, but I have no experience/no Weaviate instance so I had not tested it with the "real" weaviate - I updated the tests and mocked httpx in the way I **think** it will work, but I am not 100% sure if I got it right. Actual tests with weaviate would be great (and you had done such tests before so it might be way easier for you). J. ------------- This is the summary of the changes made. Beware .... it's ..... LOOOONG..... Add Python 3.13 support for Airflow. Added Python 3.13 support across the codebase, including: * CI/CD workflow updates to handle Python 3.13. * Dockerfile and environment variable adjustments for compatibility. * Documentation updates to list Python 3.13 as supported. * Dependency and code changes for compatibility with new/changed libraries (e.g., numpy 2.x, greenlet, pendulum). * Test and plugin manager adjustments for Python 3.13-specific behaviors. * Removal or skipping of tests and features not compatible with Python 3.13. * Improved error handling and logging for unsupported plugins/providers on Python 3.13. * Updated all provider README.rst files to: * List Python 3.13 as a supported version. * Adjust dependency requirements for Python 3.13 (e.g., pandas, pyarrow, ray, etc.). * Add or update notes about excluded or conditionally supported dependencies for Python 3.13. * Document any known incompatibilities or workarounds for specific providers. * Updated all provider pyproject.toml files and automation to: * Add Python 3.13 to the supported classifiers. * Adjust requires-python and dependency constraints for Python 3.13. * Exclude Python 3.13 for providers that are not yet compatible (e.g., those depending on Flask AppBuilder). * Add conditional dependencies for new versions of libraries required by Python 3.13. * Updated Breeze and related scripts to: * Build and test with Python 3.13. * Update documentation and release scripts to include Python 3.13. * Ensure all dev tooling and test environments work with Python 3.13. * Removed remaining FAB related code from core and removed tests for older FAB provider versions (<1.3.0) * some of the tests in core still depend on FAB - but we make them optional - both for parsing (we need to skip some imports) and execution - because they test a legacy behaviour of embedding Flask plugins that is still supported in core. * k8s tests now run also with SimpleAuthManager and SimpleAuthManager is used automatically when running tests in Python 3.13 * migration tests had to be excluded for Python 3.13 because we cannot migrate down on Python 3.13 without FAB tables created. The tests will be brought back when FAB is supported or when we release 3.1 and start working on 3.2, we should be able to migrate down to 3.1 * Weaviate tests have been updated to handle case where weaviate-client is > 2.10.0 - because it started to use httpx instead of requests, and Python 3.13 actually unblocks using older version of weaviate (it was previously blocked indirectly via Flask dependencies and grpcio). * Added doc change in our rules on what is the "default" Python version for airflow images - handling the case where we cannot use "newest" Python version as default where not all "regular" image providers are supported in the latest version J. On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote: > I also started to do more preparation and extracting things that we will > need to merge first and release providers with some limits lifted. We need > to make our providers released in PyPI prepared upfront because currently > we are not able to generate PyPI constraints for Python 3.13 - I already > have one of those merged - https://github.com/apache/airflow/pull/51994 > where we have to exclude `ray` from Python 3.13 and release provider before > we get the PR "green" - similarly I have just created > https://github.com/apache/airflow/pull/52060 to all providers that are > currently limiting pandas (unnecessary it seems from my tests in 3.13 PR). > > J. > > > On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <amoghdesai....@gmail.com> > wrote: > >> Thanks for taking this on Jarek. >> >> These are not easy tasks to work on and I cannot think of a better person >> to do it than you! >> >> I will review it shortly this week. >> >> I think it is a good thing to introduce in Airflow 3.1 and I would not be >> too much in favour >> to cherry pick it to the 3.x series, just for convenience sake. >> >> Thanks & Regards, >> Amogh Desai >> >> >> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > Hey here, >> > >> > Over the last few days and weeks I made significant progress with Python >> > 3.13 support and while the PR is not yet fully green - we are getting >> > close. I undrafted my PR recently: >> > >> > https://github.com/apache/airflow/pull/46891 >> > >> > I have a kind request for others to start reviewing it, I will be happy >> to >> > respond to questions and I am also happy to accept some fixups to solve >> > things in a better way if possible, I did a few hacks and left some >> TODOs >> > that might need some creative ideas to be solved in a better, simpler >> way. >> > >> > I split the PR into several commits - and keep it continuously rebased >> on >> > top of the latest main. I had to remove some providers (including FAB - >> > which is actually cool, because we can finally see that airflow >> codebase is >> > finally working even without FAB being around) - some of them are yet to >> > implement Python 3.13 (for example Apache Beam is working on it) so we >> > might add them back soon. >> > >> > Note - Python 3.13 was a WAY more difficult migration than Python 3.12 - >> > there were a number of things removed/changed and small behavioral >> changes >> > that we relied on, also the dependencies are catching up way later, and >> > there are some tricky dependencies of ours that make things more complex >> > when it comes to selecting "min" versions of those. But we are finally >> > getting there. >> > >> > Looks like we will be on-time to have 3.13 support for Airflow 3.1. We >> > might also attempt to cherry-pick it for 3.0, but it might be too much >> of a >> > hassle, so we can decide after we merge this one to main. >> > >> > J. >> > >> >
