Hi Jarek, That was quite a long PR to review. I focused most of my review time on the areas I’m more familiar with — (1) and (4). But I ended up going through the rest as well 🙂
Great job getting that PR green! I really appreciate the inline comments with active TODOS, they are helpful and make it easier to follow the context. Thanks & Regards, Amogh Desai On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote: > Hi Jarek, > > cool! Thanks for all the efforts and the breath in preparing this! Added > some comments in PR gut greatly appreciate (1) that notes describe the > upper bindings and (2) that the reference image default stays with 3.12 > as 3.13 lags critical providers being ready. > > Jens > > On 15.07.25 19:52, Jarek Potiuk wrote: > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews > and > > merge !!!! * > > > > The PR is finally about to get green - after about 100 or so iterations > (or > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the > > making). https://github.com/apache/airflow/pull/46891. I would really > > appreciate some final reviews and approvals. > > > > Jens and Pavan had already made a number of comments over the loooong > > course of this one and I **think** I addressed all of them. I am happy to > > get more comments - maybe some of the choices I made are outdated or can > be > > improved - but now having the PR green it will be way easier to iterate > on > > it > > > > It took a really long time and a number of iterations and some > opinionated > > choices. Mostly because FAB cannot be installed yet with Python 3.13 - > but > > it also gave me an opportunity to move out or run conditionally tests - > for > > all cases where FAB was still not completely removed from the core. One > of > > our tests (lowest-deps core Python 3.13) runs the complete suite of tests > > for core where `fab-provider` and `flask-app-builder` are completely > > missing from the CI environment (BTW. `uv sync` is an awesome feature of > > `uv` and it made it possible). And a number of changes from other people > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to > > be merged before we could get this one green). > > > > So I have a kind request - for those who know the codebase well - please > > take a look and let me know if you have any comments. Yes, it's a > monstrous > > PR and - unfortunately - it's not easily breakable into smaller ones, > but I > > tried to make it as easy to review as possible. I split it into 4 > > independent commits (which will still have to be squashed and merged > > together): > > > > 1) Breeze changes for Python 3.13 - changes in breeze code that enable > > Python 3.13 > > 2) Pyproject.toml changes in Providers for Python 3.13 (including > > dependencies and python_requires) - big number of files > > 3) README.rst changes for Providers for Python 3.13 (big number of files) > > 4) Actual changes that were necessary to get the PR green for Python 3.13 > > (smaller number, but more substantial changes in airflow-core and > provider > > code that makes Python 3.13 work) > > > > Currently excluded providers for Python 3.13: > > ** apache.beam* > > ** apache.kafka* > > ** fab* > > ** **yandex* > > ** ydb* > > > > They are excluded for various reasons - TODO:s left explaining the > reasons > > and we can gradually bring them in as blockers are removed). Detailed > > summary of changes in the PR and copied below. > > > > When you review the code, you can focus on the part that you are > interested > > in (1) - if you want to look at breeze and (4) if you want to look at > code > > changes, and skip rather easily absolutely necessary dependency > requirement > > changes in (2) and mostly-automated changes from (3). > > > > Yep I am asking for a lot, sorry, but I can't see how I could break that > > one down into separate PRs easily. > > > > I have two kind requests to take a closer look specifically: > > > > a) for *Pavan* -> if you could check the aws-wrangler removal replacing > it > > with botocore - for docs publishing. I am not 100% sure if I got it > right, > > I did test it but a second pair of hands would be great to test it > > > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones > > working on Waeviate in the past) -> I would really appreciate it if you > > could test the Weaviate changes I made. Weaviate was one of the > problematic > > changes because dependency update for 3.13 "freed" upgrade to a newer > > version of Waeviate client (>=4.10.0) that used a httpx rather than > > requests - and we had some low-level mock tests for retry scenarios (429, > > 500, ConnectErrors). And in order to fix it, I had to not only have code > in > > tests for mocking either httpx or requests, but I also had to change the > > retriable error detection in Weaviate provider. I **think** I made proper > > changes there, but I have no experience/no Weaviate instance so I had not > > tested it with the "real" weaviate - I updated the tests and mocked httpx > > in the way I **think** it will work, but I am not 100% sure if I got it > > right. Actual tests with weaviate would be great (and you had done such > > tests before so it might be way easier for you). > > > > J. > > > > ------------- > > > > This is the summary of the changes made. Beware .... it's ..... > > LOOOONG..... > > > > Add Python 3.13 support for Airflow. > > > > Added Python 3.13 support across the codebase, including: > > * CI/CD workflow updates to handle Python 3.13. > > * Dockerfile and environment variable adjustments for compatibility. > > * Documentation updates to list Python 3.13 as supported. > > * Dependency and code changes for compatibility with new/changed > > libraries (e.g., numpy 2.x, greenlet, pendulum). > > * Test and plugin manager adjustments for Python 3.13-specific > > behaviors. > > * Removal or skipping of tests and features not compatible with > Python > > 3.13. > > * Improved error handling and logging for unsupported > plugins/providers > > on Python 3.13. > > * Updated all provider README.rst files to: > > * List Python 3.13 as a supported version. > > * Adjust dependency requirements for Python 3.13 (e.g., pandas, > > pyarrow, ray, etc.). > > * Add or update notes about excluded or conditionally supported > > dependencies for Python 3.13. > > * Document any known incompatibilities or workarounds for specific > > providers. > > * Updated all provider pyproject.toml files and automation to: > > * Add Python 3.13 to the supported classifiers. > > * Adjust requires-python and dependency constraints for Python > 3.13. > > * Exclude Python 3.13 for providers that are not yet compatible > (e.g., > > those depending on Flask AppBuilder). > > * Add conditional dependencies for new versions of libraries > required > > by Python 3.13. > > * Updated Breeze and related scripts to: > > * Build and test with Python 3.13. > > * Update documentation and release scripts to include Python 3.13. > > * Ensure all dev tooling and test environments work with Python > 3.13. > > * Removed remaining FAB related code from core and removed tests for > > older FAB provider versions (<1.3.0) > > * some of the tests in core still depend on FAB - but we make them > > optional - both for parsing (we need to skip some imports) and > > execution - because they test a legacy behaviour of embedding > > Flask plugins that is still supported in core. > > * k8s tests now run also with SimpleAuthManager and > SimpleAuthManager > > is used automatically when running tests in Python 3.13 > > * migration tests had to be excluded for Python 3.13 because we > cannot > > migrate down on Python 3.13 without FAB tables created. The tests > > will be brought back when FAB is supported or when we release 3.1 > > and start working on 3.2, we should be able to migrate down to 3.1 > > * Weaviate tests have been updated to handle case where > weaviate-client > > is > 2.10.0 - because it started to use httpx instead of requests, > > and Python 3.13 actually unblocks using older version of weaviate > > (it was previously blocked indirectly via Flask dependencies and > > grpcio). > > * Added doc change in our rules on what is the "default" Python > version > > for airflow images - handling the case where we cannot use > "newest" > > Python version as default where not all "regular" image providers > are > > supported in the latest version > > > > J. > > > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> I also started to do more preparation and extracting things that we will > >> need to merge first and release providers with some limits lifted. We > need > >> to make our providers released in PyPI prepared upfront because > currently > >> we are not able to generate PyPI constraints for Python 3.13 - I already > >> have one of those merged - https://github.com/apache/airflow/pull/51994 > >> where we have to exclude `ray` from Python 3.13 and release provider > before > >> we get the PR "green" - similarly I have just created > >> https://github.com/apache/airflow/pull/52060 to all providers that are > >> currently limiting pandas (unnecessary it seems from my tests in 3.13 > PR). > >> > >> J. > >> > >> > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <amoghdesai....@gmail.com> > >> wrote: > >> > >>> Thanks for taking this on Jarek. > >>> > >>> These are not easy tasks to work on and I cannot think of a better > person > >>> to do it than you! > >>> > >>> I will review it shortly this week. > >>> > >>> I think it is a good thing to introduce in Airflow 3.1 and I would not > be > >>> too much in favour > >>> to cherry pick it to the 3.x series, just for convenience sake. > >>> > >>> Thanks & Regards, > >>> Amogh Desai > >>> > >>> > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >>> > >>>> Hey here, > >>>> > >>>> Over the last few days and weeks I made significant progress with > Python > >>>> 3.13 support and while the PR is not yet fully green - we are getting > >>>> close. I undrafted my PR recently: > >>>> > >>>> https://github.com/apache/airflow/pull/46891 > >>>> > >>>> I have a kind request for others to start reviewing it, I will be > happy > >>> to > >>>> respond to questions and I am also happy to accept some fixups to > solve > >>>> things in a better way if possible, I did a few hacks and left some > >>> TODOs > >>>> that might need some creative ideas to be solved in a better, simpler > >>> way. > >>>> I split the PR into several commits - and keep it continuously rebased > >>> on > >>>> top of the latest main. I had to remove some providers (including FAB > - > >>>> which is actually cool, because we can finally see that airflow > >>> codebase is > >>>> finally working even without FAB being around) - some of them are yet > to > >>>> implement Python 3.13 (for example Apache Beam is working on it) so we > >>>> might add them back soon. > >>>> > >>>> Note - Python 3.13 was a WAY more difficult migration than Python > 3.12 - > >>>> there were a number of things removed/changed and small behavioral > >>> changes > >>>> that we relied on, also the dependencies are catching up way later, > and > >>>> there are some tricky dependencies of ours that make things more > complex > >>>> when it comes to selecting "min" versions of those. But we are finally > >>>> getting there. > >>>> > >>>> Looks like we will be on-time to have 3.13 support for Airflow 3.1. We > >>>> might also attempt to cherry-pick it for 3.0, but it might be too much > >>> of a > >>>> hassle, so we can decide after we merge this one to main. > >>>> > >>>> J. > >>>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >