All right .. I addressed the next round of feedback and some investigations where things were questioned led to interesting findings :)
* interesting finding about having to flush() before you query.delete -> that was accidentally revealed by removing fab plugins initialization in Python 3.13 -> https://github.com/apache/airflow/pull/46891#discussion_r2209325342 * Likely root cause (long standing Python bug with mmap that was fixed in Python 3.13) for "too many open files" occasional issue people reported recently with our supervisor/task creation -> https://github.com/apache/airflow/pull/46891#discussion_r2209326571 Both are pretty cool findings and the second might even lead to some workarounds to improve number of opened file descriptors causing for example this one: https://github.com/apache/airflow/issues/51512 Thanks Vincent, Pavan, Amogh, Aritra, Jens for the reviews and some "Hawk-eye" findings :) ... Really cool The PR is now heading to be green back after some experiments with fixing things 'better" - and I plan to merge it before the dev call today - the "weaviate" implementation can be likely more thoroughly tested with the next provider's wave when we have an RC for weaviate provider - that will be easier to test I guess (and it anyhow impacts really only Python 3.13 installation of Weaviate provider). If anyone would like to make another pass - let me know and I will hold-on with merging. J. On Thu, Jul 17, 2025 at 12:03 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Cool. I have been spending more time with my family visiting me, so did > not look yet at the comments - but I will - shortly - and will be happy to > merge it soon :D > > On Wed, Jul 16, 2025 at 9:51 AM Aritra Basu <aritrabasu1...@gmail.com> > wrote: > >> Great job on the PR Jarek! Looked through the broad strokes of the PR and >> all looked good to me. >> -- >> Regards, >> Aritra Basu >> >> On Wed, 16 Jul 2025, 11:38 am Amogh Desai, <amoghdesai....@gmail.com> >> wrote: >> >> > Hi Jarek, >> > >> > That was quite a long PR to review. I focused most of my review time on >> the >> > areas >> > I’m more familiar with — (1) and (4). But I ended up going through the >> rest >> > as well 🙂 >> > >> > Great job getting that PR green! I really appreciate the inline comments >> > with active TODOS, >> > they are helpful and make it easier to follow the context. >> > >> > Thanks & Regards, >> > Amogh Desai >> > >> > >> > On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler >> <j_scheff...@gmx.de.invalid >> > > >> > wrote: >> > >> > > Hi Jarek, >> > > >> > > cool! Thanks for all the efforts and the breath in preparing this! >> Added >> > > some comments in PR gut greatly appreciate (1) that notes describe the >> > > upper bindings and (2) that the reference image default stays with >> 3.12 >> > > as 3.13 lags critical providers being ready. >> > > >> > > Jens >> > > >> > > On 15.07.25 19:52, Jarek Potiuk wrote: >> > > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final >> reviews >> > > and >> > > > merge !!!! * >> > > > >> > > > The PR is finally about to get green - after about 100 or so >> iterations >> > > (or >> > > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the >> > > > making). https://github.com/apache/airflow/pull/46891. I would >> really >> > > > appreciate some final reviews and approvals. >> > > > >> > > > Jens and Pavan had already made a number of comments over the >> loooong >> > > > course of this one and I **think** I addressed all of them. I am >> happy >> > to >> > > > get more comments - maybe some of the choices I made are outdated or >> > can >> > > be >> > > > improved - but now having the PR green it will be way easier to >> iterate >> > > on >> > > > it >> > > > >> > > > It took a really long time and a number of iterations and some >> > > opinionated >> > > > choices. Mostly because FAB cannot be installed yet with Python >> 3.13 - >> > > but >> > > > it also gave me an opportunity to move out or run conditionally >> tests - >> > > for >> > > > all cases where FAB was still not completely removed from the core. >> One >> > > of >> > > > our tests (lowest-deps core Python 3.13) runs the complete suite of >> > tests >> > > > for core where `fab-provider` and `flask-app-builder` are completely >> > > > missing from the CI environment (BTW. `uv sync` is an awesome >> feature >> > of >> > > > `uv` and it made it possible). And a number of changes from other >> > people >> > > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others >> had >> > to >> > > > be merged before we could get this one green). >> > > > >> > > > So I have a kind request - for those who know the codebase well - >> > please >> > > > take a look and let me know if you have any comments. Yes, it's a >> > > monstrous >> > > > PR and - unfortunately - it's not easily breakable into smaller >> ones, >> > > but I >> > > > tried to make it as easy to review as possible. I split it into 4 >> > > > independent commits (which will still have to be squashed and merged >> > > > together): >> > > > >> > > > 1) Breeze changes for Python 3.13 - changes in breeze code that >> enable >> > > > Python 3.13 >> > > > 2) Pyproject.toml changes in Providers for Python 3.13 (including >> > > > dependencies and python_requires) - big number of files >> > > > 3) README.rst changes for Providers for Python 3.13 (big number of >> > files) >> > > > 4) Actual changes that were necessary to get the PR green for Python >> > 3.13 >> > > > (smaller number, but more substantial changes in airflow-core and >> > > provider >> > > > code that makes Python 3.13 work) >> > > > >> > > > Currently excluded providers for Python 3.13: >> > > > ** apache.beam* >> > > > ** apache.kafka* >> > > > ** fab* >> > > > ** **yandex* >> > > > ** ydb* >> > > > >> > > > They are excluded for various reasons - TODO:s left explaining the >> > > reasons >> > > > and we can gradually bring them in as blockers are removed). >> Detailed >> > > > summary of changes in the PR and copied below. >> > > > >> > > > When you review the code, you can focus on the part that you are >> > > interested >> > > > in (1) - if you want to look at breeze and (4) if you want to look >> at >> > > code >> > > > changes, and skip rather easily absolutely necessary dependency >> > > requirement >> > > > changes in (2) and mostly-automated changes from (3). >> > > > >> > > > Yep I am asking for a lot, sorry, but I can't see how I could break >> > that >> > > > one down into separate PRs easily. >> > > > >> > > > I have two kind requests to take a closer look specifically: >> > > > >> > > > a) for *Pavan* -> if you could check the aws-wrangler removal >> replacing >> > > it >> > > > with botocore - for docs publishing. I am not 100% sure if I got it >> > > right, >> > > > I did test it but a second pair of hands would be great to test it >> > > > >> > > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the >> ones >> > > > working on Waeviate in the past) -> I would really appreciate it if >> you >> > > > could test the Weaviate changes I made. Weaviate was one of the >> > > problematic >> > > > changes because dependency update for 3.13 "freed" upgrade to a >> newer >> > > > version of Waeviate client (>=4.10.0) that used a httpx rather than >> > > > requests - and we had some low-level mock tests for retry scenarios >> > (429, >> > > > 500, ConnectErrors). And in order to fix it, I had to not only have >> > code >> > > in >> > > > tests for mocking either httpx or requests, but I also had to change >> > the >> > > > retriable error detection in Weaviate provider. I **think** I made >> > proper >> > > > changes there, but I have no experience/no Weaviate instance so I >> had >> > not >> > > > tested it with the "real" weaviate - I updated the tests and mocked >> > httpx >> > > > in the way I **think** it will work, but I am not 100% sure if I >> got it >> > > > right. Actual tests with weaviate would be great (and you had done >> such >> > > > tests before so it might be way easier for you). >> > > > >> > > > J. >> > > > >> > > > ------------- >> > > > >> > > > This is the summary of the changes made. Beware .... it's ..... >> > > > LOOOONG..... >> > > > >> > > > Add Python 3.13 support for Airflow. >> > > > >> > > > Added Python 3.13 support across the codebase, including: >> > > > * CI/CD workflow updates to handle Python 3.13. >> > > > * Dockerfile and environment variable adjustments for >> > compatibility. >> > > > * Documentation updates to list Python 3.13 as supported. >> > > > * Dependency and code changes for compatibility with >> new/changed >> > > > libraries (e.g., numpy 2.x, greenlet, pendulum). >> > > > * Test and plugin manager adjustments for Python 3.13-specific >> > > > behaviors. >> > > > * Removal or skipping of tests and features not compatible with >> > > Python >> > > > 3.13. >> > > > * Improved error handling and logging for unsupported >> > > plugins/providers >> > > > on Python 3.13. >> > > > * Updated all provider README.rst files to: >> > > > * List Python 3.13 as a supported version. >> > > > * Adjust dependency requirements for Python 3.13 (e.g., >> pandas, >> > > > pyarrow, ray, etc.). >> > > > * Add or update notes about excluded or conditionally >> supported >> > > > dependencies for Python 3.13. >> > > > * Document any known incompatibilities or workarounds for >> > specific >> > > > providers. >> > > > * Updated all provider pyproject.toml files and automation to: >> > > > * Add Python 3.13 to the supported classifiers. >> > > > * Adjust requires-python and dependency constraints for >> Python >> > > 3.13. >> > > > * Exclude Python 3.13 for providers that are not yet >> compatible >> > > (e.g., >> > > > those depending on Flask AppBuilder). >> > > > * Add conditional dependencies for new versions of libraries >> > > required >> > > > by Python 3.13. >> > > > * Updated Breeze and related scripts to: >> > > > * Build and test with Python 3.13. >> > > > * Update documentation and release scripts to include Python >> > 3.13. >> > > > * Ensure all dev tooling and test environments work with >> Python >> > > 3.13. >> > > > * Removed remaining FAB related code from core and removed >> tests >> > for >> > > > older FAB provider versions (<1.3.0) >> > > > * some of the tests in core still depend on FAB - but we make >> them >> > > > optional - both for parsing (we need to skip some imports) >> and >> > > > execution - because they test a legacy behaviour of embedding >> > > > Flask plugins that is still supported in core. >> > > > * k8s tests now run also with SimpleAuthManager and >> > > SimpleAuthManager >> > > > is used automatically when running tests in Python 3.13 >> > > > * migration tests had to be excluded for Python 3.13 because we >> > > cannot >> > > > migrate down on Python 3.13 without FAB tables created. The >> > tests >> > > > will be brought back when FAB is supported or when we release >> > 3.1 >> > > > and start working on 3.2, we should be able to migrate down >> to >> > 3.1 >> > > > * Weaviate tests have been updated to handle case where >> > > weaviate-client >> > > > is > 2.10.0 - because it started to use httpx instead of >> > requests, >> > > > and Python 3.13 actually unblocks using older version of >> > weaviate >> > > > (it was previously blocked indirectly via Flask dependencies >> and >> > > > grpcio). >> > > > * Added doc change in our rules on what is the "default" Python >> > > version >> > > > for airflow images - handling the case where we cannot use >> > > "newest" >> > > > Python version as default where not all "regular" image >> > providers >> > > are >> > > > supported in the latest version >> > > > >> > > > J. >> > > > >> > > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> >> wrote: >> > > > >> > > >> I also started to do more preparation and extracting things that we >> > will >> > > >> need to merge first and release providers with some limits lifted. >> We >> > > need >> > > >> to make our providers released in PyPI prepared upfront because >> > > currently >> > > >> we are not able to generate PyPI constraints for Python 3.13 - I >> > already >> > > >> have one of those merged - >> > https://github.com/apache/airflow/pull/51994 >> > > >> where we have to exclude `ray` from Python 3.13 and release >> provider >> > > before >> > > >> we get the PR "green" - similarly I have just created >> > > >> https://github.com/apache/airflow/pull/52060 to all providers that >> > are >> > > >> currently limiting pandas (unnecessary it seems from my tests in >> 3.13 >> > > PR). >> > > >> >> > > >> J. >> > > >> >> > > >> >> > > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai < >> amoghdesai....@gmail.com >> > > >> > > >> wrote: >> > > >> >> > > >>> Thanks for taking this on Jarek. >> > > >>> >> > > >>> These are not easy tasks to work on and I cannot think of a better >> > > person >> > > >>> to do it than you! >> > > >>> >> > > >>> I will review it shortly this week. >> > > >>> >> > > >>> I think it is a good thing to introduce in Airflow 3.1 and I would >> > not >> > > be >> > > >>> too much in favour >> > > >>> to cherry pick it to the 3.x series, just for convenience sake. >> > > >>> >> > > >>> Thanks & Regards, >> > > >>> Amogh Desai >> > > >>> >> > > >>> >> > > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> >> > wrote: >> > > >>> >> > > >>>> Hey here, >> > > >>>> >> > > >>>> Over the last few days and weeks I made significant progress with >> > > Python >> > > >>>> 3.13 support and while the PR is not yet fully green - we are >> > getting >> > > >>>> close. I undrafted my PR recently: >> > > >>>> >> > > >>>> https://github.com/apache/airflow/pull/46891 >> > > >>>> >> > > >>>> I have a kind request for others to start reviewing it, I will be >> > > happy >> > > >>> to >> > > >>>> respond to questions and I am also happy to accept some fixups to >> > > solve >> > > >>>> things in a better way if possible, I did a few hacks and left >> some >> > > >>> TODOs >> > > >>>> that might need some creative ideas to be solved in a better, >> > simpler >> > > >>> way. >> > > >>>> I split the PR into several commits - and keep it continuously >> > rebased >> > > >>> on >> > > >>>> top of the latest main. I had to remove some providers (including >> > FAB >> > > - >> > > >>>> which is actually cool, because we can finally see that airflow >> > > >>> codebase is >> > > >>>> finally working even without FAB being around) - some of them are >> > yet >> > > to >> > > >>>> implement Python 3.13 (for example Apache Beam is working on it) >> so >> > we >> > > >>>> might add them back soon. >> > > >>>> >> > > >>>> Note - Python 3.13 was a WAY more difficult migration than Python >> > > 3.12 - >> > > >>>> there were a number of things removed/changed and small >> behavioral >> > > >>> changes >> > > >>>> that we relied on, also the dependencies are catching up way >> later, >> > > and >> > > >>>> there are some tricky dependencies of ours that make things more >> > > complex >> > > >>>> when it comes to selecting "min" versions of those. But we are >> > finally >> > > >>>> getting there. >> > > >>>> >> > > >>>> Looks like we will be on-time to have 3.13 support for Airflow >> 3.1. >> > We >> > > >>>> might also attempt to cherry-pick it for 3.0, but it might be too >> > much >> > > >>> of a >> > > >>>> hassle, so we can decide after we merge this one to main. >> > > >>>> >> > > >>>> J. >> > > >>>> >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > > For additional commands, e-mail: dev-h...@airflow.apache.org >> > > >> > > >> > >> >
