Re: [DISCUSS] Tests structure for providers

> If you look for it, you will find several variants of the "Test pyramid" with anything from 3 to 7 layers and with names that are sometimes the same but they are different places in different pyramids - sometimes even the same names reverted in different variants. Yeah, it's kind of like the "de

Re: [DISCUSS] Tests structure for providers

> Basically, I like the idea and we'll have to put a little time into splitting all the exiting not-system-tests out into unit and integ. Is that more or less what you are thinking? Actually - not really :). We should change (and this is planned) all the provider's tests to "non-DB" tests - so m

Re: [DISCUSS] Mark Multi Exec Config as stable and remove old hardcoded hybrid execs

> To Niko: I was not aware that there are special cases for hybrid executors in > the core. I'd really propose to clean them up for Airflow 3. Because there is > an alternative. And I assume the gap in Helm chart is almost closed, just > missing a release. Thanks Jens, glad to have raised aware

Re: Airflow should deprecate the term "DAG" for end users

Yep. that will do as well for me - in the form proposed. We do not "have to" redefine the name, leaving it as simply "dag" and explaining the origin while clearly separating from it is also a good way to segway to "somewhat cyclic" workflow / graph if we decide to (which I think will happen sooner

Re: [DISCUSS] Tests structure for providers

I guess I'm missing what would go in the provider's integration tests then, but I'll catch up once I see it in action. - ferruzzi From: Jarek Potiuk Sent: Tuesday, February 18, 2025 3:34 PM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSS] Tests structu

Re: Airflow should deprecate the term "DAG" for end users

Also, to hopefully at a little more humor to our "lively debate" > I’m not sure conflating your opinion on this with “it’s just good engineering to make things blurry sometimes” creates a logical truth. I think it really depends on the definition of "engineer", for me "engineer" is someone who b

Re: [DISCUSS] Tests structure for providers

> I guess I'm missing what would go in the provider's integration tests then, but I'll catch up once I see it in action. Tl;DR; Look it up - there are quite a few of those already. Take a look at apache/kafka for example. Generally speaking for providers, what we are testing is not "internals" of

Re: Airflow should deprecate the term "DAG" for end users

I stayed out of this discussion for the longest time, but I really try to avoid "naming" discussions. However, the extensions needed to Airflow for enhanced AI capabilities such as cyclic operations and "human in the loop" workflows is something I have been thinking a fair bit about. I also agree

Re: Airflow should deprecate the term "DAG" for end users

Long after opening this can of worms, I also agree with Daniel S: Let's define "DAG" in the context of Airflow and be done with it (at least for now :) ). I've opened a docs PR attempting to do just that: https://github.com/apache/airflow/pull/46875 O

Re: Airflow should deprecate the term "DAG" for end users

My two shillings: I came to Airflow knowing what a DAG was in the math sense, and I was a bit surprised to see it used for Airflow. Our DAGS aren't technically DAGs and haven't been since task retries were introduced, maybe even before that. I'd support what Daniel said. IFF we're going t

Re: Airflow should deprecate the term "DAG" for end users

> -0.5 > Even though the new UI prefers "Dag", I don't think that means "DAG" is incorrect. I think we have a bunch of related things discussed here - I am not exactly sure what the -0.5 in this context means Brent. Would it be possible to clarify ? On Tue, Feb 18, 2025 at 6:01 PM Ambika Garg w

Re: [DISCUSS] Time to say goodbye to the old UI?

HUGE +1! That would have multiple benefits: - Forcing people to use the new UI will help us to identity bugs/gaps in the new UI - There is still quite a bit of code in Airflow codebase (especially in auth managers) to support old UI. Removing the old UI would allow us to clean that up On 2025/02

Re: [DISCUSS] Time to say goodbye to the old UI?

+1 to this, great step. Regards, Pavan On Tue, Feb 18, 2025, 16:26 Buğra Öztürk wrote: > +1. It would be a great step! It will enable us to clean more and reduce > duplicate code/usage and effort. > > On Tue, Feb 18, 2025 at 5:20 PM Brent Bovenzi > > wrote: > > > +1 > > > > - We absolutely don

Re: Airflow should deprecate the term "DAG" for end users

+1 would vote for Airflow's *D*ynamic *A*ction *G*raph (DAG) *Dynamic*: Reflects the capability of Airflow to handle workflows that aren’t strictly acyclic, but can adapt to changes, iterations and loops within broader context of workflow. *Action*: Emphasizes that airflow is fundamentally about

Re: [DISCUSS] Time to say goodbye to the old UI?

+1 on making the new UI by default as soon as possible since there will be more feedback and testing. The old UI features can be ported unless a blocker similar to how the Airflow 2 UI itself was ported to React incrementally in each release from Flask templates. Airflow 3 UI in React and Chakra 3

Re: Airflow should deprecate the term "DAG" for end users

I’m not sure conflating your opinion on this with “it’s just good engineering to make things blurry sometimes” creates a logical truth. :-) On Tue, Feb 18, 2025 at 12:41 PM Jarek Potiuk wrote: > Oh no.. Swords, swords, :) > > Hello Brian ! Nice to see you back in the conversations. We've b

Re: [DISCUSS] Time to say goodbye to the old UI?

+1 - We absolutely don't want to have the old UI when 3.0 launches. - It's a big change that can't wait until the last minute - There are enough changes in main that the old UI is now breaking Please check out our UI meta issue for getting to featur

Re: [LAZY CONSENSUS] Remove caplog usage from Unit Tests

There will be of course few exceptions we have to use it otherwise the functionality isn't tested properly. Those exceptions will be allowed if the maintainer decides and the test falls into those few instances. Adding a label will allow to merge of the PR. We should discourage otherwise and tell t

Re: Airflow should deprecate the term "DAG" for end users

I love "Dynamic Airflow Graph" +10 to that Constance. And yes. I fully agree with your "somewhat" acyclic statement. I saw people mentioning "event-driven" workflows to better describe the AI/ML, but that for me is a synonym of "completely unmanageable workflows" (similar to micro-services) - what

Re: [DISCUSS] Turn "tests_common" into separate distribution for development

> One thing I would like to avoid is having the `[test]` extra show up in the released packages though (it’s not important, just would be nice if we can avoid that) That is already not happening (and won't, all those devel extras in dependencies (e is gone at the moment I added hatchling, it handl

Re: [DISCUSS] Turn "tests_common" into separate distribution for development

BTW. After reading the proposed structure again python-subfolder -> that is the biggest point to discuss it, and I would rather (at least now) first move airflow_core to a sub-directory, not necessarily move all "python" projects to a sub-folder. There is very little value in such grouping, and eve

Re: Airflow should deprecate the term "DAG" for end users

First of all - yes, agree with Lee and Ash, I think we do not need to change the public interface, I'd also be -0 on that, but docs, internal strings, yes we could. But After thinking a bit and looking at the discussion here, where people have concerns, I have an alternative take. And I know

Re: Airflow should deprecate the term "DAG" for end users

Sorry for the few typos ... My slight dyslexia did not help and autocorrect did not help either this time : dept -> debt, routed -> rooted and a few other small typos :) . On Tue, Feb 18, 2025 at 4:28 PM Jarek Potiuk wrote: > First of all - yes, agree with Lee and Ash, I think we do not need to

Re: [DISCUSS] Time to say goodbye to the old UI?

+10 :) -- I can't wait to see all the old UI code being deleted. On Tue, 18 Feb 2025 at 21:25, Vikram Koka wrote: > Thanks for starting this thread, Jed. > > Around September last year, I was pessimistic about when this would happen. > I have been very impressed by our rate of progress on th

Re: [DISCUSS] Time to say goodbye to the old UI?

Well, I'd say we are all on the same page then. Thanks everyone. Brent, let it rip!

Re: [DISCUSS] Time to say goodbye to the old UI?

Would removing it imply that there's not going to be Airflow 2.12 for sure? Do we want to limit ourselves this way? On Tue, Feb 18, 2025 at 12:35 PM Abhishek Bhakat wrote: > Where do I find the docs for plugins with the new web UI alternative for < > > https://airflow.apache.org/docs/apache-airf

Re: [DISCUSS] Time to say goodbye to the old UI?

Draft: https://github.com/apache/airflow/pull/46871 On Tue, Feb 18, 2025 at 2:26 PM Ferruzzi, Dennis wrote: > +1, burn it down. You folks have put so much work into the new UI, I'm > sure it will be good to see it get its due attention from the users and > this feels like the perfect time. > >

Re: [DISCUSS] Time to say goodbye to the old UI?

Thanks for starting this thread, Jed. Around September last year, I was pessimistic about when this would happen. I have been very impressed by our rate of progress on this set of AIPs over the last few months. I see the visible progress even across the alpha releases and am really pleased by wher

Re: Airflow should deprecate the term "DAG" for end users

Hey all, I appreciate the discussion about how we position the concept of a DAG in Airflow. While I agree with updating documentation to de-emphasize the strict mathematical definition, I do not think we should change the public interface to use "dag" instead of "DAG" everywhere. It adds unnecessa

Re: [DISCUSS] Time to say goodbye to the old UI?

Huge +1000. NOW! 3 (and even 4) reasons : * The sooner we do it, the better, It's been holding us back and making quite a few things more difficult. I had to - for example - fix quite a few tests when moving stuff around and with more moves coming, this one will make our lives easier. I even remo

Re: [DISCUSS] Time to say goodbye to the old UI?

+1. It would be a great step! It will enable us to clean more and reduce duplicate code/usage and effort. On Tue, Feb 18, 2025 at 5:20 PM Brent Bovenzi wrote: > +1 > > - We absolutely don't want to have the old UI when 3.0 launches. > - It's a big change that can't wait until the last minute > -

Re: [DISCUSS] Time to say goodbye to the old UI?

+1 I completely agree that now is the right time to remove the old UI. Keeping both UIs around at this stage is more of a drag than a safety net and is increasing the maintenance cost (things are starting to become very wonky in the old UI). Having a clean break sets clear expectations: Airflow 3

Re: [DISCUSS] Time to say goodbye to the old UI?

> Well, I'd say we are all on the same page then. Thanks everyone. Brent, let it rip! Or rather burn, Hernán, burn :D J On Tue, Feb 18, 2025 at 5:40 PM Jed Cunningham wrote: > Well, I'd say we are all on the same page then. Thanks everyone. Brent, let > it rip! >

Re: Airflow should deprecate the term "DAG" for end users

-0.5 Even though the new UI prefers "Dag", I don't think that means "DAG" is incorrect. On Tue, Feb 18, 2025 at 11:36 AM Jarek Potiuk wrote: > I love "Dynamic Airflow Graph" +10 to that Constance. > > And yes. I fully agree with your "somewhat" acyclic statement. I saw people > mentioning "even

Re: [DISCUSS] Time to say goodbye to the old UI?

+1, burn it down. You folks have put so much work into the new UI, I'm sure it will be good to see it get its due attention from the users and this feels like the perfect time. - ferruzzi From: Karthikeyan Sent: Tuesday, February 18, 2025 8:51 AM To: dev@ai

Re: [DISCUSS] Tests structure for providers

Looks like this was discussed, decided, and done over the weekend, but just wanted to say thanks for that. The redundancy in he import paths has been bugging me LOL I'm glad you landed on the unit/integration/system naming convention, I like that but I'm curious what would go in the integratio

Re: Airflow should deprecate the term "DAG" for end users

Yeah I also disagree with code changes here. This thread went in an unexpected direction since I last poked my head in :) My thought is just in docs I would de-emphasize the mathy part of this. We can say a DAG is airflow's model for a collection of tasks that run, typically on a schedule. We c

Re: Airflow should deprecate the term "DAG" for end users

damnit --- meant to say is *not* strictly speaking On Tue, Feb 18, 2025 at 10:46 AM Daniel Standish < daniel.stand...@astronomer.io> wrote: > Yeah I also disagree with code changes here. This thread went in an > unexpected direction since I last poked my head in :) > > My thought is just i

Re: Airflow should deprecate the term "DAG" for end users

Ech. I would love so much if we could correct sent email same way we can correct messages in Slack :D On Tue, Feb 18, 2025 at 7:50 PM Daniel Standish wrote: > damnit --- meant to say is *not* strictly speaking > > On Tue, Feb 18, 2025 at 10:46 AM Daniel Standish < > daniel.stand...@astrono

Re: Airflow should deprecate the term "DAG" for end users

Oh no.. Swords, swords, :) Hello Brian ! Nice to see you back in the conversations. We've been discussing things like this before a few times even over the phone, I recall. Yes, I think it's partially marketing, I agree. And marketing that is important for the project, which we as engineers,

Re: Airflow should deprecate the term "DAG" for end users

I disagree with renaming it, or removing the logical idea of directed acyclic graph, or redefining the acronym. It’s less “math” to me, and more “data structure”, and a lot of things work the way they do because a dag IS a DAG. What’s the problem with calling something EXACTLY what it is? So rar

Re: [DISCUSS] Tests structure for providers

> I'm glad you landed on the unit/integration/system naming convention, I like that but I'm curious what would go in the integration path. Before I started with Airflow, I had been taught that integration test are what we call system tests. I've come to use the two terms more or less interchangea

Re: Airflow should deprecate the term "DAG" for end users

Wow what a discussion thread. Was reading it and...: I am okay to clean up docs and agree to the others that we should NOT change code interfaces. For the marketing part I need to repeat: (Almost) Everybody touching Airflow needs an explanaition what "DAG" means. Changing the acronym to have ano

Re: [DISCUSS] Time to say goodbye to the old UI?

Where do I find the docs for plugins with the new web UI alternative for < https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-view-plugin.html >? Read the AIP-68 (plugin interface), but is it in a ready state yet. - Avi On Tue, Feb 18, 2025 at 5:26 AM Ankit Chaurasia wrote: > +1

Re: [DISCUSS] Time to say goodbye to the old UI?

This is only talking about the main branch and Airflow 3.0, Airflow 2 is already branched off and that UI won’t be removed in 2.x at all. > On 18 Feb 2025, at 11:54, Michał Modras > wrote: > > Would removing it imply that there's not going to be Airflow 2.12 for sure? > Do we want to limit ou

Re: [DISCUSS] Time to say goodbye to the old UI?

Thanks Ash. Then it makes sense to me. wt., 18 lut 2025, 13:51 użytkownik Ash Berlin-Taylor napisał: > This is only talking about the main branch and Airflow 3.0, Airflow 2 is > already branched off and that UI won’t be removed in 2.x at all. > > > On 18 Feb 2025, at 11:54, Michał Modras > wrot

Re: [DISCUSS] Time to say goodbye to the old UI?

AIP-68 really is about expanding what can be done from a plugin. Main already supports fastapi_apps and extra links from plugins. Existing plugins using flask_blueprint and appbuilder_views are deprecated but will continue to work with the FAB provider installed (having that compatibility layer is

Re: [DISCUSS] Time to say goodbye to the old UI?

The OLD UI has UI Customizations which is not there in the main branch yet. I guess it would worthwhile to wait till that get's added to main branch. Could be helpful for people trying to port their plugins for new UI. On Tue, Feb 18, 2025 at 12:55 PM Michał Modras wrote: > Thanks Ash. Then it

Re: Airflow should deprecate the term "DAG" for end users

For user-facing things (e.g., `from airflow import DAG`), I’m more like -0. But for documentation, docstring, internal things and etc., we probably could still change most of them? Best, Wei > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor wrote: > > I also don’t personally think it’s worth t

Re: [DISCUSS] Turn "tests_common" into separate distribution for development

One thing I would like to avoid is having the `[test]` extra show up in the released packages though (it’s not important, just would be nice if we can avoid that) I wonder if now is also the time to move all the python code under a sub-folder So something like this Python_modules/ Airflow_co

Re: [LAZY CONSENSUS] Remove caplog usage from Unit Tests

Agreed, testing logs should be a last resort, and we likely to over-use it, but there are at least a few instances where we do explicitly want to test that certain info is included in the logs for users. -a > On 16 Feb 2025, at 11:16, Jens Scheffler wrote: > > Besides the instabilities we sa

Re: Airflow should deprecate the term "DAG" for end users

I also don’t personally think it’s worth the pain (not to mention backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on that. -ash > On 18 Feb 2025, at 04:40, Daniel Imberman wrote: > > I think my biggest concern is a marketing one and not a technical one. > > As has been mentioned o

Re: Airflow should deprecate the term "DAG" for end users

When I started contributing to Airflow, I had to read up on DAG, and now I know what it means. I wonder if we are about to have users who use the term Dag without knowing that there's a DAG. Slightly concerned. -ephraim On Tue, 18 Feb 2025 at 10:26, Wei Lee wrote: > For user-facing things (e.g