I hope more voices will be coming soon as well :). Constance - few words of explanation, for my "remove" things. I do not say we "have to" remove those things or even that we "should".
My main motivation for mentioning the list is that we often forget that when we have a new major version of software - removing stuff is as important as adding. So I think when it comes to final decisions about Airflow 3 we should be well aware about both: what we add and what we remove. So my main point is - "removal" of things should be as important a part in our future discussions as "adding", and I'd say with 10 years of Airflow history, it's more important to remove things than to add them. Side comment: Ash for one mentioned at multiple occasions how proud he is that his contributions to Airflow are net-negative - he removed more code than added. Unfortunately it's not shown any more here https://github.com/apache/airflow/graphs/contributors - because we have more than 10.000 commits, but I was hoping one day when we remove providers from main development, I can do it personally and beat Ash to it. Just wanted to stress a few things on why I think "discussing potential removals" is important: * I think it is super important to deliver Airflow 3 really fast. It took > 1.5 year to release Airflow 2 and it was far too long and keeping Airflow 1 and 2 in parallel was a huge pain and drain and it slowed us down enormously. We should absolutely limit the time when we actively develop both Airflow 2 features and Airflow 3 features. * I think most of the current deprecations of the 100+ we have does not slow us down AT ALL. Removal of those is mostly cosmetic change, a little bit clutter to remove but they have little-to-no impact on actual speed of development, it's just an old code that keeps on being around * on the other hand - some of the things I mentioned ARE slowing us down A LOT and will continue to do so until we remove them. For example, I believe "postgresql + mysql + sqlite complete versioning solution for all the possible variants of storage" will be quite a bit more complex and testing will take far more time than if we drop mysql and choose a single storage approach for Airflow 3.0. I have no hard data to back it up, but my gut feeling is that it can take at least twice as long to get it out in the hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for all the options we have there. * the telemetry will be cool - but even if we add it for 2.10, the first time meaningful data will be available is maybe 2 years from now when we do Airflow 4. For now we can completely forget about it because it's only going to show us some early adopters of Airflow 2.10. But we have surveys + we can collect data from Astronomer, Google. Amazon for some usage and identify who will be early adopters of Airflow 3 and target them first. * the fact that we will drop something for Airflow 3.0, does not mean that we have to drop it "forever". We can safely assume that Airflow 3.0 will be only used by early adopters, and many of our users will wait for 3.1, 3.2 or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those "slower moving users" from the 3.0. We can re-add things in 3.1 building on foundations of 3.0 after it is in the hands of those early adopters. I personally think we should prioritize speed of delivery of 3.0 over "complete support of every deployment option of Airflow 2" - to allow **some** of our early adopter users to migrate quickly, and add what's missing for those who would like to move later in later versions. * I think - this is the most important "product management" decision here - based on the data we have available. Which users do we target for Airflow 3.0, and what we can drop to deliver it faster (and possibly re-add more stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0 initiative and the most important decision from the Product management side that will impact the timeline of Airflow 3.0. While a lot depends on how much time and effort individuals will spend on implementing things, the decision of what we drop will impact everyone's speed and overall delivery of Airflow 3.0. It's a decision we should not take lightly. So my main point is: let's add "what we remove" as a very important point in the product discussions we are going to have. J. On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar <rbish...@amazon.com.invalid> wrote: > Like Constance and many others, there was time needed to process this > fantastical summary and approach to Airflow 3. __ > A lot of the points raised by Jarek make sense, ie being prescriptive on > what features we want to be there on Day 1 vs launched on 3.x - this does > allow us to move quickly for our users. > In line with Constance's concern, we need to be mindful of what we decide > to keep and what we are willing to cut - which could be a painstaking > process that could offset any gains of trying to be faster. I also believe > that bringing all these new amazing features on Airflow 3 will peak the > interest of early adopters and eventually get others interested in > migration. However, I believe this migration will be a slow process and > will present a gap in certain functionalities that users may want before > entertaining any move to Airflow 3. There are still a lot of folks using > v1.10 today. There were several tactical initiatives in the past few months > with intent on bringing new functionality, ie Multi-team, to Airflow 2.x > and I feel that while these efforts are not wasted, there should still be > an option to continue improving Airflow 2 to avoid alienating our users on > the basis of a future promise in Airflow 3, that may not be easy to migrate > towards. > > -- Rajesh > > > > > > > > > dev@airflow.apache.org <mailto:dev@airflow.apache.org> < > dev@airflow.apache.org <mailto:dev@airflow.apache.org>> > > > On 2024-05-07, 12:26 PM, "Constance Martineau" > <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > <mailto:consta...@astronomer.io.inva > <mailto:consta...@astronomer.io.inva>>LID> > wrote: > > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > > > > > > > Thank you Jarek for the detailed input. I've taken some time to digest your > points before responding. > > > > > You've outlined a bold vision for Airflow 3, and I agree that being > decisive about the features and architectures will be the key to success. > However, before we make final decisions on what features to cut or retain, > it would be beneficial to have a more comprehensive understanding of how > the current features are utilized by the open-source community. > > > > > @Kaxil Naik <ka...@astronomer.io <mailto:ka...@astronomer.io> <mailto: > ka...@astronomer.io <mailto:ka...@astronomer.io>>> recently initiated a > discussion on > collecting telemetry from open-source deployments: > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> < > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>>. > This data > could be critical in ensuring our decisions are well-informed and reflect > the real-world usage patterns of our users, not just those from managed > environments like Astro, MWAA or GCC. > > > > > It's essential that we challenge our assumptions and base our decisions on > a holistic view of feature usage. Identifying potential cuts is a critical > step, but let's ensure our strategy aligns with the needs and preferences > of the broader Airflow community. > > > > > On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com <mailto: > ja...@potiuk.com> <mailto:ja...@potiuk.com <mailto:ja...@potiuk.com>>> > wrote: > > > > > > I am currently on sick leave, and still recovering - hoping to be able to > > travel next week to the US as planned, so I just wanted to break out of > it > > to make one comment here. > > > > I got a clearer head now a bit with medications hopefully working. I am > > still taking it that should help me to get over the current state, and I > > wanted to take a look at this discussion unraveling first. Over last week > > I disconnected from "day-to-day" Airflow and put some thoughts (as much > as > > I could in my current state) on it. The whole subject of this thread was > > started from that - how the current discussions on AIP-67 and others > change > > if we consider Airflow 3 is "starting". > > > > The price for back-compat is speed of development and quality. More > > combinations to test, more unexpected issues uncovered, necessity to keep > > parallel paths (old/new) while adding new features. All what Constance > > wrote about and what Ash explained. We already started to trip over our > own > > feet mutliple times in a few last releases. Have we tested all > combinations > > of deployment in Airflow 2.8 and 2.9 - not really, I think we already see > > that in a number of "combos" of features things are not working in as > > stable a way as they did before. > > > > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long > > time (or even move out) as they will not want to move to Airflow 3. A lot > > of the work implemented in AIP-44 and design of AIP-67 was done around > > back-compatibility. but yes - > > it would have been way easier if designed anew without back-compatibility > > in mind. And if we implement it and release it in Airflow 2 it will make > > new Airflow feature development even harder. That's why I wanted to treat > > it as "tactical" solution - hoping that in Airflow 3 we can make it > > "properly" - and that's why I started the discussion here when I sensed > > that we are "close" to Airflow 3 discussion, because I wanted to see what > > options we have there. This is why I have not yet concluded voting on > > AIP-67 waiting for the result of this discussion here. > > > > But if we are ready to go for Airflow.3 then I'd say there are two > > important things that should be part of the vision. > > > > 1) *We should be far more opinionated and have far fewer options of > > running things in Airflow 3*. Even an order of magnitude more > opinionated. > > Make choices, stick to it, perfect those opinionated choices to suit > 80/20 > > (or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting > > the 20% that might choose to stay at Airflow 2. We can choose now which > > ~20% of cases we do not want to handle deliberately. And we should be > very, > > very strict about it. Default should be "no choice". This will radically > > simplify deployment and should make it easier to simplify Airflow > > development and DAG authoring experience because we will have less cases > to > > support. Even if we plan to add more options in the future, the first > > version of Airflow 3 should support one deployment approach only. This is > > the only way we can deliver it fast. And we should be very bold there. > > Choose one option and go for it in pretty much every place we have > choices > > now. We should Aim for Airflow 3.0 to support only a subset of current > > users - but those who are most likely to migrate first and those with the > > biggest need for the new features. We can think 3.x to support more > cases, > > but 3.0 should be as opinionated as humanly possible. > > > > And this deployment option should be also something ALL our stakeholders > > will feel OK with as a way forward in their offering. > > > > My candidates (and yes, some are bold): > > > > * *Drop MySQL*. If we have a single thing that makes us avoid our schema > > and DB migration - this is the case. Let's choose Postgres 15+ and use > some > > of the great features there. This will also enable much faster async SQL > > implementation and a number of other optimisations - not to mention > cutting > > every single change in development and testing time by literally half. > And > > we should not look back to adding MySQL. > > * *Drop Celery/Sequential Executor* and start with Local + K8S only (and > > AWS/Google others can continue developing theirs of course in parallel > and > > continue Hybrid executor work). Later - we figure out a better solution > to > > support "small" tasks using some new K8S features and possibly non-k8s > > solutions (Ray-based?) > > * *Cut Connection and Variable Management from DB/UI*. Leave only Secrets > > Management. Later when we have a 100% extensible React UI, we can add a > > "local DB secrets manager" add-on > > * *Choose a single way for DAG storage that will support versioning from > > day one*. Bear in mind we can add others later. Bolke's idea of using > > FSspec is an interesting one, we should see if it is feasible. > > * *Drop FAB completely (including custom plugins) and invest in > > implementing Auth Manager based on a dedicated, external solution* > > (KeyCloak > > that we've discussed before as a likely candidate) > > * *Leave Providers with Airflow 2 and add tests to make sure they are > > Airflow 3 future-compatible *- develop a way where we continue > development > > and contributions for Providers with Airflow 2 and add complete tests to > > run them with Airflow 3. This way we can continue developing Provider > > features independently, and make them work for Airflow 2 (and continue > > adding features for Airflow 2 users alongside Airflow 2 bugfixes), while > > also gradually fix any Airflow3 incompatibilities and instead of > > "back-compatibility" tests make provider "forward-compatibility" tests so > > that future Providers are tested and work on Airflow 3. Also it will make > > it easiest to continue Airflow 2 (bugfixes) + Providers tested without > > investing in changing the current CI / test harness. > > * *Simplify Test Harness for Airflow 3 from the start *- without > providers > > and 790+ dependencies, we could vastly simplify Airflow3 testing > (basically > > make CI jobs from scratch) using mostly standard Python tooling (while we > > can continue making use of the current test harness for Airflow 2 + > > Providers and extend it with Airflow 3 future-compatibility tests). That > > means Breeze would be only staying in Airflow 2 + Providers repo as we > > should be able to achieve most of what we have there with local venv/ > > tooling (especially with uv as underlying tooling). > > > > 2) *I think we only add very few new "important" features. *Absolute > > minimum to make Airflow 3 appealing and add them only in Airflow 3: > > versioning, multi-team, pluggable UI should only be Airflow 3 - it makes > no > > sense to invest into Airflow 2 if we already know Airflow 3 is coming - > > that generally triples effort needed to get them out. We should drop new > > features development in Airflow 2. This will give users incentive to move > > to 3 if the new features will be worth it. Even paying > > compatibility/migration price. > > > > Versionig, for example: I believe if we decide to go only with Airflow 3 > > and cut some of the above (Postgres only, Single versioning DAG storage) > we > > can make bolder decisions in versioning and support simpler models from > the > > get go (and deliver it faster). And we should add only a few - but > > important - features that our users clearly asked for and focus on > > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or > 2.11). > > Similarly - multi-team can be simplified if we cut things from the list > > above and have Task isolation as first-class citizens in Airflow (and the > > only option). > > > > My candidates very much concur with the list shared by Kaxil in the doc + > > I'd add multi-team (but simplified thanks to the cuts). But I also here > > would mostly revert to Astronomer, Google. AWS team to define > collectively > > what is the absolute minimum set of features that would get the "target" > > part of their customers happy. And ONLY do that. > > > > So in short - I think the big part of our discussion should be what we > are > > ready to drop when we start airflow 3 and be very bold. Once we know we > > should figure out the absolute minimum of things that we can add that > will > > benefit a significant part of our users (and make use of increased speed > > because we dropped things). > > > > J. > > > > > > On Mon, May 6, 2024 at 8:40 PM Constance Martineau > > <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva> > <mailto:consta...@astronomer.io.inva > <mailto:consta...@astronomer.io.inva>>lid> > wrote: > > > > > Hi Michal, > > > > > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate your > > > concerns about the migration overhead for our users with a major new > > > version and see the appeal in your suggestion to integrate many of the > > > proposed changes into Airflow 2 through separate AIPs. It’s a valid > point > > > and certainly aligns with the value of making incremental improvements. > > > > > > However, after looking closely at the enhancements outlined for Airflow > > 3, > > > I'm convinced they warrant a new major release. Here’s why: > > > > > > 1. *Core Architectural Changes:* We’re looking at foundational changes > > > with Airflow 3—like redefining task priorities, separating task > > > definition > > > and task execution, and new AIPs like DAG versioning. remote execution > > > and restricting database access from workers. These aren’t just > > > incremental > > > improvements but major shifts that will set the stage for the next > > > decade > > > of Airflow’s architecture. Grouping these changes into a major release > > > will > > > help us make these transitions more cleanly and with fewer constraints > > > from > > > past decisions. > > > 2. *Code Clean-Up*: Our main branch has accumulated over 140 > > deprecated > > > issues, and this will only grow if we continue without a major > > cleanup. > > > This makes it increasingly difficult to implement new features > > > effectively > > > while maintaining backward compatibility. A major release allows us to > > > address these issues head-on, reducing technical debt and paving the > > way > > > for a more robust platform. > > > 3. *Managing Breaking Changes:* Let’s take the example of restricting > > > database access from workers. It’s a necessary move for better > > security > > > and > > > also potentially scalability reasons (reduces DB load). Many users > > have > > > workflows that interact with the DB, either by using raw sql or by > > > leveraging a session object. We could implement this feature in > > Airflow > > > 2 > > > and avoid breaking existing workflows by continuing to have the old > > > standard mode as default - much of the work is already done - but that > > > would mean supporting both the new secure mode and the old standard > > mode > > > indefinitely and design new features with the assumption that most > > will > > > continue using the old standard mode. With Airflow 3, we can make > > secure > > > mode the default or even the only option, simplifying implementation > > and > > > future development. This is just one example where it is feasible to > > > implement in Airflow 2, but is better if we release it under the > > > context of > > > Airflow 3. > > > 4. *Future-Proofing for New Features:* Airflow 3 will open up > > > possibilities for handling workflows beyond batch processing. Features > > > like > > > real-time DAG execution through API and multi-language task support > > are > > > big > > > steps forward, significantly expanding Airflow’s utility. > > > > > > > > > While integrating these updates into Airflow 2 might look less > disruptive > > > initially, the scale and nature of the required changes really support > a > > > move to Airflow 3. It’s not just about adding new features; it’s about > > > setting up Airflow so that it continues to remain relevant for the next > > ten > > > years. > > > > > > Constance > > > > > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <a...@apache.org > <mailto:a...@apache.org> <mailto:a...@apache.org <mailto:a...@apache.org>>> > wrote: > > > > > > > There's a lot of technical debt hiding in Airflow, especially the > > > > scheduler that makes it harder and harder to efficiently add new > > > features. > > > > > > > > At some point, very soon, we are going to have to remove some very > > > > infrequently used back compat shims that negatively affect > performance. > > > > Without doing that the pace at which we can realistically add some of > > the > > > > more exciting features tends towards zero. Developer speed of > > > contributors > > > > is a factor here too! > > > > > > > > So while we are still using SemVer, that necessitates v3. > > > > > > > > Ash > > > > > > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmod...@google.com > <mailto:michalmod...@google.com> <mailto:michalmod...@google.com <mailto: > michalmod...@google.com>> > > > .INVALID> > > > > wrote: > > > > >+1 to Jens's & Bolke's points here and in the doc > > > > > > > > > >I agree we should work on clarifying the directions we would like > > > Airflow > > > > >to go. Introducing a new major Airflow version is a massive overhead > > for > > > > >users, who would need to plan for migrations, onboarding the new > > Airflow > > > > >(with a slightly different architecture), etc., and effectively > > Airflow > > > 2 > > > > >would live in parallel for a long time. > > > > > > > > > >Personally, I think most of the points in Kaxil's/Vikram's doc are > > > > valuable > > > > >projects of their own, and I could imagine all of them being > delivered > > > as > > > > >separate AIPs within Airflow 2 (surely new minor versions of Airflow > > > 2). I > > > > >am not sure if the scope of changes and the goal we want to achieve > is > > > a) > > > > >clear enough b) broad enough to call for a new major version. > > > > > > > > > >Best, > > > > >Michal > > > > > > > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T) > > > > ><jens.scheff...@de.bosch.com.inva <mailto: > jens.scheff...@de.bosch.com.inva> <mailto:jens.scheff...@de.bosch.com.inva > <mailto:jens.scheff...@de.bosch.com.inva>>lid> wrote: > > > > > > > > > >> Thanks for the document write-up, Kaxil. I assume this is mostly a > > > > vision > > > > >> statement. > > > > >> > > > > >> Looking forward for a larger addendum where we can collect things > > that > > > > we > > > > >> all can vote and agree on as targets. > > > > >> > > > > >> As I started earlier with a confluence page and it seems this is > not > > > > >> accessible to all, shall we convert this to a Google Doc for > better > > > > >> collaboration and item collection? > > > > >> > > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> < > https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> < > https://aka.ms/o0ukef&gt;>> > > > > >> ________________________________ > > > > >> From: Vikram Koka <vik...@astronomer.io.inva <mailto: > vik...@astronomer.io.inva> <mailto:vik...@astronomer.io.inva <mailto: > vik...@astronomer.io.inva>>LID> > > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM > > > > >> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > <mailto:dev@airflow.apache.org <mailto:dev@airflow.apache.org>> < > dev@airflow.apache.org <mailto:dev@airflow.apache.org> <mailto: > dev@airflow.apache.org <mailto:dev@airflow.apache.org>>> > > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) > vs > > > > >> strategic (Airflow 3) approach > > > > >> > > > > >> Thank you for your feedback, Bolke and Andrey! > > > > >> > > > > >> Bolke, > > > > >> I have replied to some of your comments in the doc. > > > > >> I will provide a detailed write up on the "Interactive DAG run" > (or > > > > >> synchronous DAG run) capability, which has generated some early > > > > questions. > > > > >> I had intended to get an AIP published for that as a follow-up, > but > > I > > > > >> believe that a simpler write up would be useful ahead of the AIP. > > > > >> > > > > >> Andrey, > > > > >> You raise an interesting point. > > > > >> > > > > >> As part of the Airflow 2.0 release, we as a community had decided > to > > > > >> strictly adhere to Semver as detailed in the document you > > referenced. > > > We > > > > >> also consciously split out the "Core Airflow" releases from the > > > > "Provider" > > > > >> releases at that time. We had a clear expectation then for the > > cadence > > > > of > > > > >> both minor and patch releases, which we have generally adhered to > > > since > > > > >> then. > > > > >> > > > > >> Personally, I am more concerned about our Provider releases right > > now, > > > > as > > > > >> compared to the cadence of our major releases. I believe that one > of > > > the > > > > >> proposed changes in the Airflow 3 document i.e. the clear > separation > > > for > > > > >> Task Execution will help here, but more may be needed. > > > > >> > > > > >> Definitely interested in more feedback on this as well. > > > > >> > > > > >> Vikram > > > > >> > > > > >> > > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin < > > > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is> <mailto: > andrey.ans...@taragol.is <mailto:andrey.ans...@taragol.is>> > > > > > > > > > >> wrote: > > > > >> > > > > >> > I would like to propose to change (at least discuss) release > > policy > > > > >> around > > > > >> > the Major version of Airflow. > > > > >> > > > > > >> > Right now it is described as "These releases do not happen with > > any > > > > >> regular > > > > >> > interval or on any predictable schedule." : > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0> > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0> > < > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> > ;> > > > > >> < > > > > >> > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > < > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> > ;> > > > > >> > > > > > >> > > > > > >> > So maybe it is time to make it schedulable, e.g. one per two > years > > > or > > > > so. > > > > >> > This one could help us to avoid such a discussion in the future, > > > like > > > > "We > > > > >> > don't know when Airflow 4 is coming.". At the moment when the > new > > > > major > > > > >> > version will be released new features wouldn't be added in the > old > > > > major > > > > >> > version, however we would support bug / security for a while, > > e.g. 1 > > > > year > > > > >> > for bug fixes, 3 years for security fixes with a total 5 year > > > > lifecycle > > > > >> per > > > > >> > a major version. These just are approximate time periods for a > > > > definition > > > > >> > of current period, bugfix period and security fix period. > > > > >> > > > > > >> > In contributors' perspective it helps with dropping the > deprecated > > > > stuff > > > > >> > which resolves some old problem: we have to support everything > > > > including > > > > >> > deprecated stuff and without schedulable lifecycle for the > > > deprecated > > > > >> stuff > > > > >> > it could be showstopper for the new feature, because sometimes > it > > > > hard to > > > > >> > support two different approaches for long period of time with no > > > hope > > > > >> that > > > > >> > it will happen soon. For some fundamental stuff which do not > > > require a > > > > >> lot > > > > >> > things time to support we could postponed removal for next after > > the > > > > next > > > > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5 > > > > >> > > > > > >> > In the user perspective, they have at least bug fix support for > a > > > > while, > > > > >> if > > > > >> > someone want to use legacy version it their choice, however no > new > > > > >> > features, no new version of providers (after one year) > > > > >> > > > > > >> > > > > > >> > ---- > > > > >> > Best Wishes > > > > >> > *Andrey Anshin* > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bdbr...@gmail.com > <mailto:bdbr...@gmail.com> <mailto:bdbr...@gmail.com <mailto: > bdbr...@gmail.com>>> > > > > wrote: > > > > >> > > > > > >> > > I have left several comments :-). And on interactive dag runs > > even > > > > >> after > > > > >> > > the explanation of Vikram I still don't have a clue what we > want > > > to > > > > >> > > accomplish there :-P. > > > > >> > > > > > > >> > > I would like to see a mantra or team for Airflow 3. That helps > > > > nudging > > > > >> > > people in the same direction. Suggestions in the comments. > > > > >> > > > > > > >> > > Bolke > > > > >> > > Sent from my iPhone > > > > >> > > > > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka > > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva> <mailto: > vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>>lid > > > > > > > > > >> > > wrote: > > > > >> > > > > > > > >> > > > Good point Jed. > > > > >> > > > I responded back to your comment in the doc as well and very > > > open > > > > to > > > > >> > > > changing the term in the doc. > > > > >> > > > > > > > >> > > > Used the term "interactive DAG run" as the ability to invoke > > or > > > > >> > trigger a > > > > >> > > > DAG run through the API, with the expectation of getting > back > > a > > > > >> result > > > > >> > > > immediately. An alternate term could be a "synchronous DAG > > run". > > > > >> > > > > > > > >> > > > Regardless, this is a significant change so a good term to > > > > indicate > > > > >> the > > > > >> > > > expansion from "batch runs only" is warranted. Very open to > > > > different > > > > >> > > terms > > > > >> > > > here. > > > > >> > > > > > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham < > > > > >> > jedcunning...@apache.org <mailto:jedcunning...@apache.org> > <mailto:jedcunning...@apache.org <mailto:jedcunning...@apache.org>> > > > > >> > > > > > > > >> > > >> wrote: > > > > >> > > >> > > > > >> > > >> Very exciting! Looks like we will have a busy period of > time > > > > ahead > > > > >> of > > > > >> > > us. > > > > >> > > >> Overall I like the plan so far, especially using this > year's > > > > Airflow > > > > >> > > Summit > > > > >> > > >> as an opportunity to announce and gather feedback, and the > > 2025 > > > > >> > version > > > > >> > > to > > > > >> > > >> pitch upgrading. > > > > >> > > >> > > > > >> > > >> I left a comment in the doc, but we might want to iterate > on > > > the > > > > >> > > >> terminology we use for high priority or "synchronous" DAG > > runs > > > to > > > > >> > serve > > > > >> > > LLM > > > > >> > > >> responses - I find "interactive DAG runs" a bit confusing. > > > > >> > > >> > > > > >> > > > > > > >> > > > > > > --------------------------------------------------------------------- > > > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > <mailto:dev-unsubscr...@airflow.apache.org> <mailto: > dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org>> > > > > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org > <mailto:dev-h...@airflow.apache.org> <mailto:dev-h...@airflow.apache.org > <mailto:dev-h...@airflow.apache.org>> > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > >
