For user-facing things (e.g., `from airflow import DAG`), I’m more like -0. But for documentation, docstring, internal things and etc., we probably could still change most of them?
Best, Wei > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor <a...@apache.org> wrote: > > I also don’t personally think it’s worth the pain (not to mention backcompat > workaround) to rename DAG to Dag, so I’d be -0.5 on that. > > -ash > >> On 18 Feb 2025, at 04:40, Daniel Imberman <daniel.imber...@gmail.com> wrote: >> >> I think my biggest concern is a marketing one and not a technical one. >> >> As has been mentioned on the thread the terms airflow and dag are kind of >> synonymous and I certainly don’t want to give the impression that we are >> breaking more than we are breaking. >> >> I wouldn’t die on this hill, but I’m slightly concerned. >> >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> wrote: >> >>> I’m not sure about adding ruff rules here 🤔 I think ruff rules are best >>> suited for user-facing things but not the airflow code base itself. If what >>> we mean is adding a rule to avoid users using "DAG" *after* we rename it, >>> it's definitely a +1000. >>> >>> I just created GitHub issues for this removing "DAG" >>> >>> https://github.com/apache/airflow/issues/46842 >>> https://github.com/apache/airflow/issues/46843 >>> >>> One thing I'm not sure about is whether we want to get rid of `from >>> airflow import DAG` as well. 🤔 >>> >>> Best, >>> Wei >>> >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: >>>> Hard to say until it's looked at :) >>>> >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu <aritrabasu1...@gmail.com> >>>> wrote: >>>> >>>>> I can take it up, it's mostly just a doc update right? Or are we doing >>> code >>>>> files replacement too? >>>>> -- >>>>> Regards, >>>>> Aritra Basu >>>>> >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, <ja...@potiuk.com> wrote: >>>>> >>>>>> Sounds like another +1000 files big PR is coming :) ? Any volunteers >>> to >>>>>> make it ? It's fun. >>>>>> >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P <droiddev5...@gmail.com> >>> wrote: >>>>>> >>>>>>> +1 for ruff rules :) >>>>>>> >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the >>> Airflow >>>>>>> docs, in line with the new UI changes and to make the renaming >>>>>>> consistent across user-facing pages. >>>>>>> >>>>>>> Regards, >>>>>>> Omkar >>>>>>> >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> >>> wrote: >>>>>>> >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) >>>>>>>> >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> >>> wrote: >>>>>>>> >>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag" >>>>> instead >>>>>>> of >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the >>>>> codebase >>>>>>> with >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be >>> 🤔), >>>>>>> should >>>>>>>> we >>>>>>>>> at least avoid introducing new "DAG" in the following PRs? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Wei >>>>>>>>> >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: >>>>>>>>>> https://github.com/apache/airflow/pull/43325 >>>>>>>>>> Let me know if that's the direction we want to go. >>>>>>>>>> >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is >>> synonymous >>>>>> with >>>>>>>>>>> workflows in Airflow through the way we talk and explain >>> what >>>>>>> Airflow >>>>>>>>> is >>>>>>>>>>> all about. Although folks would ask, I don’t ever use the >>>>>>>> mathematical >>>>>>>>>>> definition of DAG. >>>>>>>>>>> It will be challenging and possibly confusing for many >>> users >>>>>> making >>>>>>>>> such a >>>>>>>>>>> change - I would rather direct that energy to appending the >>>>>> Oxford >>>>>>>>>>> definition of "dag" to include a reference to workflows in >>>>>> Airflow. >>>>>>>> __ >>>>>>>>>>> >>>>>>>>>>> -- Rajesh >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com >>>>>> <mailto: >>>>>>>>>>> ja...@potiuk.com>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> CAUTION: This email originated from outside of the >>>>> organization. >>>>>> Do >>>>>>>> not >>>>>>>>>>> click links or open attachments unless you can confirm the >>>>> sender >>>>>>> and >>>>>>>>> know >>>>>>>>>>> the content is safe. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un >>>>> expéditeur >>>>>>>>> externe. >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe >>> si >>>>> vous >>>>>>> ne >>>>>>>>> pouvez >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes >>> pas >>>>>>>> certain >>>>>>>>> que >>>>>>>>>>> le contenu ne présente aucun risque. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> From Guido's post: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> "Around this time the renaming seems to have been renamed". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Naming is hard. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> J. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < >>>>> droiddev5...@gmail.com >>>>>>>>> <mailto: >>>>>>>>>>> droiddev5...@gmail.com>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Yes, completely agree with above comments, dag is an >>> Airflow >>>>>> term >>>>>>>> now >>>>>>>>>>>> rather >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not, >>> I've >>>>>>>> worked >>>>>>>>> with >>>>>>>>>>>> work >>>>>>>>>>>> folks who've been using dags in Airflow for years now >>> and are >>>>>> pro >>>>>>>>> devs >>>>>>>>>>> but >>>>>>>>>>>> have trouble remembering the full form of a DAG! For >>> them, >>>>> dag >>>>>> = >>>>>>>>> Airflow. >>>>>>>>>>>> >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but >>> will be >>>>> a >>>>>>>> major >>>>>>>>>>>> behavior >>>>>>>>>>>> change for the developer community, so we'll need a solid >>>>> plan >>>>>> on >>>>>>>>> how to >>>>>>>>>>>> introduce it, when we do. >>>>>>>>>>>> >>>>>>>>>>>> While we discuss this, I'd like to share about the Great >>>>>> Renaming >>>>>>>> in >>>>>>>>> core >>>>>>>>>>>> Python started by Guido back in 2009. We could probably >>> get >>>>>> some >>>>>>>>>>> learnings >>>>>>>>>>>> from there? Who knows! >>>>>>>>>>>> >>>>>>>>>>>> Guido's blog (2009): >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html >>>>>>>>>>> < >>>>>>>>> >>>>>>> >>>>> >>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html >>>>>>>>> >>>>>>>>>>>> Follow-up discussion (2024): >>>>>>>>>>>> >>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082 >>>>>>> < >>>>>>>>>>> >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082> >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Omkar >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < >>> weilee...@gmail.com >>>>>>>>> <mailto: >>>>>>>>>>> weilee...@gmail.com>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think we should probably just accept it as an airflow >>>>> term. >>>>>>> At >>>>>>>>> least, >>>>>>>>>>>>> that’s how I understood it when I first used Airflow. I >>>>> feel >>>>>>>>> renaming >>>>>>>>>>> it >>>>>>>>>>>> at >>>>>>>>>>>>> this stage would require considerable effort from >>>>> maintainers >>>>>>> and >>>>>>>>>>>> existing >>>>>>>>>>>>> users without providing equivalent benefits. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Wei >>>>>>>>>>>>> >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < >>>>>> kaxiln...@gmail.com >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle >>>>> this >>>>>>>>> discussion >>>>>>>>>>>> for >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much >>>>>>> disruption >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: >>>>>>>>> consta...@astronomer.io.inva>lid> >>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow >>>>>> experience >>>>>>>>> what a >>>>>>>>>>>> dag >>>>>>>>>>>>> is, >>>>>>>>>>>>>>> they'll start talking about workflow attributes - >>> stuff >>>>>> like >>>>>>>>> dags >>>>>>>>>>>> being >>>>>>>>>>>>> a >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure >>>>>> doesn't >>>>>>>>> come up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to >>> embrace >>>>> the >>>>>>> name >>>>>>>>> and >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: >>>>> vik...@astronomer.io.inva >>>>>>>>> lid> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> It's an interesting discussion and I remember >>>>> struggling >>>>>>> with >>>>>>>>> this >>>>>>>>>>>>> when I >>>>>>>>>>>>>>>> started working with Airflow. >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an >>>>>>>> established >>>>>>>>>>>> concept >>>>>>>>>>>>>>> now >>>>>>>>>>>>>>>> regardless of the origin. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am personally leaning towards the perspective >>> best >>>>>>>> expressed >>>>>>>>> by >>>>>>>>>>>>> Daniel >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather >>> than >>>>>> the >>>>>>>>> computer >>>>>>>>>>>>>>> science >>>>>>>>>>>>>>>> concept. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>> Vikram >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: >>>>>> oniko...@amazon.com.inva >>>>>>>>> lid >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're >>> right >>>>>> Ryan, >>>>>>>> DAG >>>>>>>>>>> isn't >>>>>>>>>>>>>>> great >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will >>> cause >>>>> much >>>>>>>> more >>>>>>>>>>>> wreckage >>>>>>>>>>>>>>>> than >>>>>>>>>>>>>>>>> it solves. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from >>>>> defining >>>>>>>> DAG. >>>>>>>>> I >>>>>>>>>>>> think >>>>>>>>>>>>>>>>> we've been naturally doing that as a community >>> for a >>>>>> while >>>>>>>> now >>>>>>>>>>>> anyway, >>>>>>>>>>>>>>> so >>>>>>>>>>>>>>>>> that feels like a natural step. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>> Niko >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ________________________________ >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: >>>>>>>>> a...@apache.org>> >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: >>>>>> dev@airflow.apache.org >>>>>>>> >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the >>> term >>>>>> "DAG" >>>>>>>>> for end >>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the >>>>>>>>> organization. >>>>>>>>>>> Do >>>>>>>>>>>>> not >>>>>>>>>>>>>>>>> click links or open attachments unless you can >>> confirm >>>>>> the >>>>>>>>> sender >>>>>>>>>>>> and >>>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>>> the content is safe. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient >>> d’un >>>>>>>>> expéditeur >>>>>>>>>>>>>>> externe. >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce >>>>>> jointe >>>>>>> si >>>>>>>>> vous >>>>>>>>>>> ne >>>>>>>>>>>>>>>> pouvez >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si >>> vous >>>>>> n’êtes >>>>>>>> pas >>>>>>>>>>>> certain >>>>>>>>>>>>>>>> que >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a >>> term — >>>>>>>> getting >>>>>>>>> to >>>>>>>>>>>>> re-use >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, both >>>>> sides >>>>>>> have >>>>>>>>> good >>>>>>>>>>>>>>> reasons >>>>>>>>>>>>>>>>> presented. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -a >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes >>> from, >>>>>>> "well, >>>>>>>>> no >>>>>>>>>>> one >>>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>> knows but..." and then make something up. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < >>>>>>>>> ja...@potiuk.com >>>>>>>>>>> <mailto:ja...@potiuk.com>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is >>> the >>>>>>>>>>> tongue-twister, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < >>>>>>>>> ja...@potiuk.com >>>>>>>>>>> <mailto:ja...@potiuk.com>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I >>> would >>>>> very >>>>>>>> much >>>>>>>>> want >>>>>>>>>>>> to >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>> able >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it >>> further. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk >>> it's a >>>>>>>>>>>> tongue-twister, >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to >>> pronounce >>>>>> it >>>>>>>>>>> properly. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> J. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: >>>>>>>> br...@astronomer.io.inva >>>>>>>>>> lid> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when >>>>> starting >>>>>> on >>>>>>>>> AIP-38 >>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> modernize >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more >>>>>>>>> descriptive of >>>>>>>>>>>> what >>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph >>> is an >>>>>>>>>>> implementation >>>>>>>>>>>>>>>>> detail. >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this >>> point >>>>>> "DAG" >>>>>>>> has >>>>>>>>>>> become >>>>>>>>>>>>>>>> "dag" >>>>>>>>>>>>>>>>>>> , a >>>>>>>>>>>>>>>>>>>>> word in its own right. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community >>>>>> discussion, >>>>>>>>> Airflow >>>>>>>>>>>>>>>> Summit >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's >>>>>> embrace >>>>>>>>> "dag". >>>>>>>>>>> A >>>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical >>>>> concept >>>>>>>>> behind >>>>>>>>>>> that >>>>>>>>>>>>>>>>> word. I >>>>>>>>>>>>>>>>>>>>> think that is much less effort than >>> refactoring so >>>>>>> much >>>>>>>>> code, >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow >>>>> questions, >>>>>>>> etc. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel >>> Standish >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot >>> of >>>>>> pain >>>>>>>> for >>>>>>>>>>>>>>>>> questionable >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the >>> idea. I >>>>>>> agree >>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> association >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less >>>>>>> invasive >>>>>>>>>>>>>>> mitigations. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to >>>>>> de-emphasize >>>>>>>> the >>>>>>>>>>> math >>>>>>>>>>>>>>> nerd >>>>>>>>>>>>>>>>>>>>> origins >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / >>> website / >>>>>> etc, >>>>>>>>> *never >>>>>>>>>>>>>>>> define* >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic >>>>>> graph. >>>>>>>>> Always >>>>>>>>>>>>>>> define >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is >>> like a >>>>>>>>> historical >>>>>>>>>>>>>>>>> footnote, >>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere >>> hidden. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the >>> docs >>>>> in >>>>>>>>> general >>>>>>>>>>> e.g. >>>>>>>>>>>>>>>>>>> writing >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / >>> "DAGs" >>>>>> etc. >>>>>>>> The >>>>>>>>>>>> upper >>>>>>>>>>>>>>>> case >>>>>>>>>>>>>>>>>>>>> part >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but >>> "dag" in >>>>>>>>> airlfow is >>>>>>>>>>>> just >>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with >>> "DAGs" >>>>> is >>>>>>> not >>>>>>>>> really >>>>>>>>>>>>>>>>>>> unhelpful. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow >>> is >>>>> its >>>>>>> own >>>>>>>>>>> thing, >>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> *not* strictly >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which >>> nobody >>>>>> knows >>>>>>>>> about >>>>>>>>>>>>>>>> anyway), >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that >>> normal >>>>>>> people >>>>>>>>>>>>>>> understand. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk >>> < >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it >>> will >>>>> be >>>>>>>>> extremely >>>>>>>>>>>>>>>>>>>>> difficult to >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it >>> will >>>>>> make >>>>>>> a >>>>>>>>> lot of >>>>>>>>>>>>>>>>>>> "google" >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not >>>>> finding >>>>>>> the >>>>>>>>> right >>>>>>>>>>>>>>>>>>> answers. >>>>>>>>>>>>>>>>>>>>>> This >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - >>> besides the >>>>>>>>> community >>>>>>>>>>> and >>>>>>>>>>>>>>>> ideas >>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of >>>>> examples, >>>>>>>>> problems >>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> solutions >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to >>> remember >>>>>> that >>>>>>>>> all the >>>>>>>>>>>> AI >>>>>>>>>>>>>>>>>>>>> trained >>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly >>> matching >>>>>>> queries >>>>>>>> of >>>>>>>>>>>> people. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily >>>>>> switch. >>>>>>>>> And if >>>>>>>>>>> we >>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline >>> instead >>>>> of >>>>>>>>> `dag` if >>>>>>>>>>>> not >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> above >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor >>> that it >>>>>>> might >>>>>>>>> cause >>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>> problems >>>>>>>>>>>>>>>>>>>>>>> than it solves. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will >>> think >>>>>>> it's >>>>>>>> a >>>>>>>>> good >>>>>>>>>>>>>>>> idea, I >>>>>>>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>> ok >>>>>>>>>>>>>>>>>>>>>>> with it. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> J, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek >>> Bhakat >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva >>> <mailto: >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less >>> sense >>>>> to >>>>>>>>> someone >>>>>>>>>>> new >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> workflow >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the >>> nature >>>>>> of >>>>>>>>> being >>>>>>>>>>>>>>> acyclic. >>>>>>>>>>>>>>>>>>>>> Sure, >>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to >>> workaround it. >>>>>>>> Still, >>>>>>>>> in >>>>>>>>>>> my >>>>>>>>>>>>>>>>>>>>> opinion, >>>>>>>>>>>>>>>>>>>>>>> there >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in >>>>>>> workflow >>>>>>>>>>>>>>>>>>> orchestration. >>>>>>>>>>>>>>>>>>>>>> Most >>>>>>>>>>>>>>>>>>>>>>>> (*if >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be >>>>> covered >>>>>>>>> using an >>>>>>>>>>>>>>>> acyclic >>>>>>>>>>>>>>>>>>>>>> manner >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the >>> idempotency. So >>>>> I >>>>>>>> would >>>>>>>>> want >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> "acyclic" >>>>>>>>>>>>>>>>>>>>>>>> word to stick. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>>>> Avi >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: >>> bernd.stroe...@kosakya.de>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an >>>>> Airflow >>>>>>> Fan; >>>>>>>>> so >>>>>>>>>>> many >>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>> ideas. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be >>>>>> replaced >>>>>>> by >>>>>>>>> the >>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>> general >>>>>>>>>>>>>>>>>>>>>>>> Term >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow >>>>>> (Petri) >>>>>>>> Net >>>>>>>>>>> (AFN) >>>>>>>>>>>>>>>>>>> (maybe >>>>>>>>>>>>>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>>>>>>>>> a direction); >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a >>>>> Graph >>>>>>>>> Database. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow >>> Graph >>>>>> (AFG) >>>>>>>>> might be >>>>>>>>>>>>>>>>>>>>> assigned >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member >>> of a >>>>>>>> library. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different >>>>> structure >>>>>>>> than >>>>>>>>> a >>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>> Graph. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: >>>>>>>>>>> bernd.stroe...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov >>> <ikholo...@google.com.inva >>>>>>>>> <mailto: >>>>>>>>>>> ikholo...@google.com.inva>LID> >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: >>>>>>>>> dev@airflow.apache.org >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the >>> term >>>>>>> "DAG" >>>>>>>>> for >>>>>>>>>>> end >>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly >>>>>> suboptimal, >>>>>>> it >>>>>>>>> is >>>>>>>>>>> part >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>> Airflow >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many >>> levels, >>>>> that >>>>>>> any >>>>>>>>>>> attempt >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not >>> reduce it. >>>>>> The >>>>>>>>> only >>>>>>>>>>>> thing >>>>>>>>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>>>>>>>>>>> worse >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is >>> when >>>>>>> there >>>>>>>>> are >>>>>>>>>>> two >>>>>>>>>>>>>>> ways >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> define >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and >>>>> tutorials >>>>>>>> will >>>>>>>>>>>> suddenly >>>>>>>>>>>>>>>>>>>>> become >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows >>> as >>>>>>> "DAG"s. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the >>> users >>>>> away >>>>>>>> with >>>>>>>>> a >>>>>>>>>>>> number >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>> breaking >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more >>>>>> breaking >>>>>>>>> changes >>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people >>> are >>>>>>>> looking >>>>>>>>> for. >>>>>>>>>>>>>>>>>>>>> Attempting >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be >>>>> interpreted >>>>>>> as >>>>>>>> an >>>>>>>>>>> even >>>>>>>>>>>>>>>>>>>>> stronger >>>>>>>>>>>>>>>>>>>>>>>> signal >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing >>> the >>>>>> term >>>>>>>> in >>>>>>>>> the >>>>>>>>>>>> long >>>>>>>>>>>>>>>>>>>>> run. I >>>>>>>>>>>>>>>>>>>>>>>> even >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating >>> it in >>>>>> the >>>>>>>> DAG >>>>>>>>>>>>>>> definition >>>>>>>>>>>>>>>>>>>>>>>> interface. >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in >>> other >>>>>> places >>>>>>>> in >>>>>>>>>>>> Airflow, >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>>>>> Igor >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas >>> Harenslak >>>>>>>>>>>>>>>>>>>>>>> <b...@astronomer.io.inva <mailto: >>>>>>>> b...@astronomer.io.inva >>>>>>>>>> lid >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of “DAG” >>> have >>>>>>> already >>>>>>>>> faded >>>>>>>>>>>> over >>>>>>>>>>>>>>>>>>>>> time, >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to >>>>>> create >>>>>>>>> cyclic >>>>>>>>>>>>>>>>>>> graphs, >>>>>>>>>>>>>>>>>>>>>> e.g. >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine >>>>> these >>>>>>>>>>> properties >>>>>>>>>>>>>>>>>>>>>> vanishing >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that >>>>>>> perspective I >>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>> changing >>>>>>>>>>>>>>>>>>>>>>>>>> “DAG" to a more generic name. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do >>>>>> naming: >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I think “workflow” is the most fitting >>> name. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this >>> change, I >>>>>>> suggest >>>>>>>>>>>> defining >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>> clear >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce >>> the >>>>>>>>> deprecation in >>>>>>>>>>>>>>>>>>>>> Airflow >>>>>>>>>>>>>>>>>>>>>> 3, >>>>>>>>>>>>>>>>>>>>>>>>>> and remove “DAG” in Airflow 4? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Bas >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < >>>>>>>> neil4r...@gmail.com >>>>>>>>>>> <mailto:neil4r...@gmail.com>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG, >>>>>>>> especially >>>>>>>>> when >>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term >>> wholeheartedly. >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or >>>>> confusing >>>>>>>>> about it >>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>> all, >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly >>>>>>> confusing >>>>>>>>> to >>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>> who >>>>>>>>>>>>>>>>>>>>>>>>>>> have been >>>>>>>>>>>>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping >>>>> Chung >>>>>>>>>>>>>>>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: >>>>>>>>> t...@astronomer.io.inva >>>>>>>>>>>> lid >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with >>> the >>>>> term >>>>>>>> DAG. >>>>>>>>> The >>>>>>>>>>>> only >>>>>>>>>>>>>>>>>>>>>>> problem >>>>>>>>>>>>>>>>>>>>>>>>>> (aside >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling people—including >>>>>>> myself—to >>>>>>>>> stop >>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> term) >>>>>>>>>>>>>>>>>>>>>>>>>> is to >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can’t recall who, but someone >>> mentioned >>>>>>>>> “workflow” is >>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>> very >>>>>>>>>>>>>>>>>>>>>>>>>> accurate >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term “definition” was >>>>>>> proposed, >>>>>>>>> but >>>>>>>>>>>> it’s a >>>>>>>>>>>>>>>>>>>>> bit >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few >>> places >>>>> and >>>>>>> kept >>>>>>>>>>> finding >>>>>>>>>>>>>>>>>>>>> myself >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting “what definition?” and >>> wanting to >>>>>>>> clarify >>>>>>>>> “DAG >>>>>>>>>>>>>>>>>>>>>>> definition” >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose). >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens >>> Scheffler >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: >>>>>>>>>>> j_scheff...@gmx.de.inva>LID> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the >>> exactly >>>>> same >>>>>>>>>>>> observation, >>>>>>>>>>>>>>>>>>>>> had a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is >>> always >>>>> an >>>>>>>>>>> introduction >>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone >>>>>>>>>>>>>>>>>>>>>>>>>> joins >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming >>> makes >>>>>>> sense. >>>>>>>>>>>>>>>>>>> Especially >>>>>>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a >>>>>>> reasonable >>>>>>>>> step. >>>>>>>>>>>>>>>>>>>>> Concepts >>>>>>>>>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding >>>>> below >>>>>>> the >>>>>>>>> desk >>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we >>> keep >>>>>>>> details >>>>>>>>> of >>>>>>>>>>>>>>>>>>> python >>>>>>>>>>>>>>>>>>>>>> names >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a >>> DAG… >>>>>> but >>>>>>>> user >>>>>>>>>>>> facing >>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>>>>>>>>> workflow. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan >>> Hatter < >>>>>>>>>>>>>>>>>>>>>>> ryan.hat...@astronomer.io <mailto: >>>>>>>>> ryan.hat...@astronomer.io >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your >>> swords... at >>>>>>> least >>>>>>>>> for >>>>>>>>>>>> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little >>> meaning to >>>>>>>> Airflow >>>>>>>>>>>> users. >>>>>>>>>>>>>>>>>>>>>> Indeed, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some >>>>>> mathematicians >>>>>>>> and >>>>>>>>>>>>>>>>>>> software >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers >>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually >>>>>> matter. >>>>>>>> For >>>>>>>>>>>> someone >>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow >>>>> orchestration, >>>>>>> one >>>>>>>>> of >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they >>>>>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a >>> DAG?" >>>>>> The >>>>>>>>> answer >>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> almost >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic >>> graph. >>>>> You >>>>>>>> don't >>>>>>>>>>> need >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> worry >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what >>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your >>>>> workflow." >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at >>>>> least >>>>>> a >>>>>>>>> couple >>>>>>>>>>>>>>>>>>>>> important >>>>>>>>>>>>>>>>>>>>>>>>> reasons: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As >>> mentioned >>>>>>> above, >>>>>>>>> "DAG" >>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and >>> confusing. >>>>>> We >>>>>>>> want >>>>>>>>>>>> Airflow >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and >>>>>>>>>>>>>>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right >>> off the >>>>>> bat >>>>>>>>> creates >>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>> initial >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow >>>>>>> Concepts*: >>>>>>>>> The >>>>>>>>>>> DAG >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow >>> workflow. The >>>>>>>>> workflow >>>>>>>>>>>>>>>>>>> includes >>>>>>>>>>>>>>>>>>>>>> its >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen >>>>> other >>>>>>>>>>> parameters, >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesn’t account for. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the >>> Airflow >>>>>>>> homepage >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < >>>>>>>>>>> https://airflow.apache.org/>>. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Airflow® is a platform >>> created by >>>>>> the >>>>>>>>>>> community >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor >>> workflows. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is >>>>> Airflow?" >>>>>>> docs >>>>>>>>> page >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html < >>>>>>>>>>> >>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , >>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what >>> Airflow is >>>>>>>> without >>>>>>>>>>> using >>>>>>>>>>>>>>>>>>>>> "DAG." >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the >>> term >>>>> is >>>>>>>>>>> introduced >>>>>>>>>>>>>>>>>>> out >>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to >>> explain >>>>> it: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a >>>>> collection >>>>>>> of >>>>>>>>> tasks >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs >>> in >>>>>> these >>>>>>>>>>>>>>>>>>> introductions >>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't >>>>>> orchestrate >>>>>>>>> DAGs; >>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>> orchestrates >>>>>>>>>>>>>>>>>>>>>>>>> workflows*. >>>>>>>>>>>>>>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons >>>>>> irrelevant >>>>>>>> to >>>>>>>>>>> almost >>>>>>>>>>>>>>>>>>>>> every >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an >>> alias >>>>> for >>>>>>> the >>>>>>>>> term >>>>>>>>>>>> "DAG" >>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace >>> "DAG" >>>>>> with >>>>>>>>>>>> "workflow". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag( >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow( >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running >>>>> away. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> >>> ------------------------------------------------------------------ >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: >>>>>>>>>>>>>>>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: >>>>>>>>>>> dev-unsubscr...@airflow.apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>>>>>> dev-h...@airflow.apache.org <mailto: >>>>>>>>>>> dev-h...@airflow.apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
