Hello here, I have been thinking a lot recently and discussing with some people and I am more and more convinced it's about the time we - as a community - should start doing changes considering "Airflow 2" current and "Airflow 3" future.
*TL;DR: I think we should seriously start work on Airflow 3 and decide what it means for our AIPs - to treat some of them as more "tactical" - things that should go into Airflow 2 and some "strategic" ones - being foundational for Airflow 3 - with different goals and criteria.* A lot of us already think that way and a lot of us have already talked about it for quite some time, so you should treat my mail mostly as a little trigger "let's start publicly discussing what it might mean for us and our community and let's make it clear about the target of the initiatives we do". Some might be surprised it comes from me as I've been often saying "no Airflow 3 without a good reason" or "possibly we will have no Airflow 3", but I think (and a number of people I spoke to have similar opinion) we have plenty of reasons to make some bold moves now. Over the last 4 years since Airflow 2 was out, a lot has changed and we have a number of different needs that current Airflow 2 cannot **really** do well - LLM/Gen-AI mainly as the important trigger - Cloud Native is the "way to go" - need to submit DAGs in other ways than dropping them to a shared DAG folder. - local testing and fast iteration on developing pipelines. - ability to run tasks with workflow with "affinity" so that they can share inputs/outputs in shared CPU/GPU memory - ability to integrate seamlessly with other workflow engines - making Airflow a "workflow of workflows - probably way more - all that while keeping a lot of the strengths of Airflow 2 - such as continuing to have the option of using the many thousands of operators with 90+ providers. All those above - we could implement better if we get rid of a number of the implicit or explicit luggage we have in Airflow 2. I think the last two proposals from Jens: AIP-68 and AIP-69 reflect very much that - both would have been much easier and straightforward if we got Airflow 3 re-designed basically at a drawing board with boldly dropping some Airflow 3 assumptions. And if we implemented core airflow 3 - taking the best part of what we have now in Airflow 2, but generally dropping the luggage in a new framework. And it won't be possible without breaking some fundamental assumptions and making Airflow 3 quite heavily incompatible with Airflow 2 >From "my" camp - dropping the need of having the 700+ dependencies for Airflow + all providers in a single Python interpreter, dropinnig dependency on Flask/Plugins/FAB would be a huge win on its own. Not mentioning being able to split provider's development and contribution from airflow core (while keeping the development of providers as well and contributions) - this has been highly requested. And I think we have a lot of people in our community who would be able (and would love) to do it - I think a number of us (including myself) are a bit burned out and tired of just maintaining things in Airflow in a backwards-compatible way and would jump on the opportunity to rebuilding Airflow. But - we of course cannot forget about Airflow 2 users. We do not want to "stop the world" for them. We want to keep fixing things and adding incremental changes - and those things do not necessarily super "future-proof". They should help to "keep the lights on" for a while - which means that in a number of cases it could be "band-aid". AIP-44 (internal-API), AIP-67 (multi-team) are more of those. So - what I think we might want to do as a community: * start working on Airflow 3 foundations (and decide what it means for our users and developer community). Decide what to keep, what to drop, what to redesign, assumptions to recreate. * explicitly split the initiatives/AIPs we have to target Airflow 2 and Airflow 3 and treat them a bit differently in terms of future-proofness I would love to hear your thoughts on that (bracing for the storm of those). J.
