It seems that our current conclusion is to use "dag" or "Dag" instead of "DAG" whenever possible. Should we replace all "DAG" in the codebase with "dag" or "Dag"? If it's too late for that (which it might be đ€), should we at least avoid introducing new "DAG" in the following PRs?
Best, Wei On 2024/10/23 17:16:55 Brent Bovenzi wrote: > Here's a PR to use "dag" as a word in the new 3.0 UI: > https://github.com/apache/airflow/pull/43325 > Let me know if that's the direction we want to go. > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar > <rbish...@amazon.com.invalid> wrote: > > > I think Brent & Daniel summarized it best, "dag" is synonymous with > > workflows in Airflow through the way we talk and explain what Airflow is > > all about. Although folks would ask, I donât ever use the mathematical > > definition of DAG. > > It will be challenging and possibly confusing for many users making such a > > change - I would rather direct that energy to appending the Oxford > > definition of "dag" to include a reference to workflows in Airflow. __ > > > > -- Rajesh > > > > > > > > > > > > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > > ja...@potiuk.com>> wrote: > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and know > > the content is safe. > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur externe. > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne pouvez > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain que > > le contenu ne prĂ©sente aucun risque. > > > > > > > > > > > > > > From Guido's post: > > > > > > "Around this time the renaming seems to have been renamed". > > > > > > Naming is hard. > > > > > > J. > > > > > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev5...@gmail.com <mailto: > > droiddev5...@gmail.com>> wrote: > > > > > > > Yes, completely agree with above comments, dag is an Airflow term now > > > rather > > > than just a "directed acyclic graph". Believe it or not, I've worked with > > > work > > > folks who've been using dags in Airflow for years now and are pro devs > > but > > > have trouble remembering the full form of a DAG! For them, dag = Airflow. > > > > > > workflow, pipeline, flow are surely better terms but will be a major > > > behavior > > > change for the developer community, so we'll need a solid plan on how to > > > introduce it, when we do. > > > > > > While we discuss this, I'd like to share about the Great Renaming in core > > > Python started by Guido back in 2009. We could probably get some > > learnings > > > from there? Who knows! > > > > > > Guido's blog (2009): > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > <https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html> > > > Follow-up discussion (2024): > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 < > > https://discuss.python.org/t/finishing-the-great-renaming/54082> > > > > > > Cheers, > > > Omkar > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee...@gmail.com <mailto: > > weilee...@gmail.com>> wrote: > > > > > > > I think we should probably just accept it as an airflow term. At least, > > > > thatâs how I understood it when I first used Airflow. I feel renaming > > it > > > at > > > > this stage would require considerable effort from maintainers and > > > existing > > > > users without providing equivalent benefits. > > > > > > > > Best, > > > > Wei > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxiln...@gmail.com > > <mailto:kaxiln...@gmail.com>> wrote: > > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this discussion > > > for > > > > > Airflow 4 :) -- but right now it will cause too much disruption > > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > > > <consta...@astronomer.io.inva > > > > > <mailto:consta...@astronomer.io.inva>lid> > > wrote: > > > > > > > > > >> In my experience, when you ask those with Airflow experience what a > > > dag > > > > is, > > > > >> they'll start talking about workflow attributes - stuff like dags > > > being > > > > a > > > > >> series of steps or tasks with owners. The structure doesn't come up. > > > > >> > > > > >> Echo-ing others, at this point, my vote is to embrace the name and > > > > >> de-emphasize the mathematical structure aspect. > > > > >> > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>lid> > > > > >> wrote: > > > > >> > > > > >>> It's an interesting discussion and I remember struggling with this > > > > when I > > > > >>> started working with Airflow. > > > > >>> But, I also agree with the viewpoint of it being an established > > > concept > > > > >> now > > > > >>> regardless of the origin. > > > > >>> > > > > >>> I am personally leaning towards the perspective best expressed by > > > > Daniel > > > > >>> Standish and Brent of using Dag as a word, rather than the computer > > > > >> science > > > > >>> concept. > > > > >>> > > > > >>> Best regards, > > > > >>> Vikram > > > > >>> > > > > >>> > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > > >> <oniko...@amazon.com.inva <mailto:oniko...@amazon.com.inva>lid > > > > >>>> > > > > >>> wrote: > > > > >>> > > > > >>>> I agree with the general sentiment of: You're right Ryan, DAG > > isn't > > > > >> great > > > > >>>> and I'd rather workflow, but changing it will cause much more > > > wreckage > > > > >>> than > > > > >>>> it solves. > > > > >>>> > > > > >>>> Also agree with the idea to just move away from defining DAG. I > > > think > > > > >>>> we've been naturally doing that as a community for a while now > > > anyway, > > > > >> so > > > > >>>> that feels like a natural step. > > > > >>>> > > > > >>>> Cheers, > > > > >>>> Niko > > > > >>>> > > > > >>>> ________________________________ > > > > >>>> From: Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > > >>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end > > > > >> users > > > > >>>> > > > > >>>> CAUTION: This email originated from outside of the organization. > > Do > > > > not > > > > >>>> click links or open attachments unless you can confirm the sender > > > and > > > > >>> know > > > > >>>> the content is safe. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur > > > > >> externe. > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous > > ne > > > > >>> pouvez > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas > > > certain > > > > >>> que > > > > >>>> le contenu ne prĂ©sente aucun risque. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> Best argument in favour of keeping âdagsâ as a term â getting to > > > > re-use > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > https://i.imgflip.com/1xhtwh.jpg> > > > > >>>> > > > > >>>> In all seriousness: I donât mind either way, both sides have good > > > > >> reasons > > > > >>>> presented. > > > > >>>> > > > > >>>> -a > > > > >>>> > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > > >>>> <daniel.stand...@astronomer.io.inva <mailto: > > daniel.stand...@astronomer.io.inva>LID> wrote: > > > > >>>>> > > > > >>>>> Yeah just say, when asked where the name comes from, "well, no > > one > > > > >>>> actually > > > > >>>>> knows but..." and then make something up. > > > > >>>>> > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <ja...@potiuk.com > > <mailto:ja...@potiuk.com>> > > > > >> wrote: > > > > >>>>> > > > > >>>>>> Just to clarify - "directed acyclic graph" is the > > tongue-twister, > > > > >>>>>> > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <ja...@potiuk.com > > <mailto:ja...@potiuk.com>> > > > > >>> wrote: > > > > >>>>>> > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very much want > > > to > > > > >> be > > > > >>>>>> able > > > > >>>>>>> to say just "dag" without explaining it further. > > > > >>>>>>> > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a > > > tongue-twister, > > > > >>> and > > > > >>>> I > > > > >>>>>>> almost stutter on trying to recall how to pronounce it > > properly. > > > > >>>>>>> > > > > >>>>>>> J. > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > > > >>>>>> <br...@astronomer.io.inva <mailto:br...@astronomer.io.inva>lid> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> I remember we explored renaming "DAG" when starting on AIP-38 > > to > > > > >>>>>> modernize > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more descriptive of > > > what > > > > >>> one > > > > >>>>>> is > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an > > implementation > > > > >>>> detail. > > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG" has > > become > > > > >>> "dag" > > > > >>>>>> , a > > > > >>>>>>>> word in its own right. > > > > >>>>>>>> > > > > >>>>>>>> Examples for "dag" are abound in community discussion, Airflow > > > > >>> Summit > > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace "dag". > > A > > > > >> user > > > > >>>>>> just > > > > >>>>>>>> needs to learn one new word vs the technical concept behind > > that > > > > >>>> word. I > > > > >>>>>>>> think that is much less effort than refactoring so much code, > > > > >>>>>>>> documentation, blog posts, stack overflow questions, etc. > > > > >>>>>>>> > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish > > > > >>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > daniel.stand...@astronomer.io.inva>lid> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain for > > > > >>>> questionable > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I agree the > > > > >>>>>>>> association > > > > >>>>>>>>> with "directed acyclic graph" is not helpful. > > > > >>>>>>>>> > > > > >>>>>>>>> And along those lines, I offer here some less invasive > > > > >> mitigations. > > > > >>>>>>>>> > > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize the > > math > > > > >> nerd > > > > >>>>>>>> origins > > > > >>>>>>>>> of the name. That is to say, in docs / website / etc, *never > > > > >>> define* > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph. Always > > > > >> define > > > > >>>> it > > > > >>>>>>>> as a > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc. > > > > >>>>>>>>> > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a historical > > > > >>>> footnote, > > > > >>>>>>>> and > > > > >>>>>>>>> we could make one mention of it somewhere hidden. > > > > >>>>>>>>> > > > > >>>>>>>>> We could also start using lowercase in the docs in general > > e.g. > > > > >>>>>> writing > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc. The > > > upper > > > > >>> case > > > > >>>>>>>> part > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in airlfow is > > > just > > > > >>> an > > > > >>>>>>>>> airflow concept and the association with "DAGs" is not really > > > > >>>>>> unhelpful. > > > > >>>>>>>>> > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its own > > thing, > > > is > > > > >>>>>>>>> *not* strictly > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows about > > > > >>> anyway), > > > > >>>>>> and > > > > >>>>>>>>> tell them what it is in simple terms that normal people > > > > >> understand. > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk < > > ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > > > > > >>>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be extremely > > > > >>>>>>>> difficult to > > > > >>>>>>>>>> get rid of it completely. Also I think it will make a lot of > > > > >>>>>> "google" > > > > >>>>>>>>>> searches and "stack overflow" searches not finding the right > > > > >>>>>> answers. > > > > >>>>>>>>> This > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the community > > and > > > > >>> ideas > > > > >>>>>>>> that > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples, problems > > and > > > > >>>>>>>> solutions > > > > >>>>>>>>>> you can so easily find (and we have to remember that all the > > > AI > > > > >>>>>>>> trained > > > > >>>>>>>>> on > > > > >>>>>>>>>> past data will be also rather poorly matching queries of > > > people. > > > > >>>>>>>>>> > > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch. And if > > we > > > > >> do > > > > >>> - > > > > >>>>>> I > > > > >>>>>>>>>> would be for using workflow or pipeline instead of `dag` if > > > not > > > > >>> the > > > > >>>>>>>> above > > > > >>>>>>>>>> reason, but I think I am here with Igor that it might cause > > > more > > > > >>>>>>>> problems > > > > >>>>>>>>>> than it solves. > > > > >>>>>>>>>> > > > > >>>>>>>>>> But I am not 100% against - if others will think it's a good > > > > >>> idea, I > > > > >>>>>>>> am > > > > >>>>>>>>> ok > > > > >>>>>>>>>> with it. > > > > >>>>>>>>>> > > > > >>>>>>>>>> J, > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat > > > > >>>>>>>>>> <abhishek.bha...@astronomer.io.inva <mailto: > > abhishek.bha...@astronomer.io.inva>lid> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to someone > > new > > > > >> to > > > > >>>>>>>>> workflow > > > > >>>>>>>>>>> orchestration. But it does also show the nature of being > > > > >> acyclic. > > > > >>>>>>>> Sure, > > > > >>>>>>>>>> as > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it. Still, in > > my > > > > >>>>>>>> opinion, > > > > >>>>>>>>>> there > > > > >>>>>>>>>>> is generally no need for cyclic behavior in workflow > > > > >>>>>> orchestration. > > > > >>>>>>>>> Most > > > > >>>>>>>>>>> (*if > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered using an > > > > >>> acyclic > > > > >>>>>>>>> manner > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I would want > > > the > > > > >>>>>>>>> "acyclic" > > > > >>>>>>>>>>> word to stick. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Regards, > > > > >>>>>>>>>>> Avi > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > bernd.stroe...@kosakya.de <mailto:bernd.stroe...@kosakya.de>> > > > > >>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow Fan; so > > many > > > > >> new > > > > >>>>>>>>> ideas. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced by the > > > more > > > > >>>>>>>> general > > > > >>>>>>>>>>> Term > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri) Net > > (AFN) > > > > >>>>>> (maybe > > > > >>>>>>>>>>> without > > > > >>>>>>>>>>>> a direction); > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph Database. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG) might be > > > > >>>>>>>> assigned > > > > >>>>>>>>> to > > > > >>>>>>>>>> an > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a library. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> A running Graph might have a different structure than a > > > > >>>>>>>> configuration > > > > >>>>>>>>>>>> Graph. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Forget that if you think it's bullshit. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Best Regards > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Bernd Ströhle > > > > >>>>>>>>>>>> M: +49 171 5357916 > > > > >>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > bernd.stroe...@gmail.com> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> -----Original Message----- > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholo...@google.com.inva <mailto: > > ikholo...@google.com.inva>LID> > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term "DAG" for > > end > > > > >>>>>> users > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, it is > > part > > > > >> of > > > > >>>>>>>>> Airflow > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that any > > attempt > > > > >> to > > > > >>>>>>>>> change > > > > >>>>>>>>>> it > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The only > > > thing > > > > >>>>>>>> that is > > > > >>>>>>>>>>> worse > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when there are > > two > > > > >> ways > > > > >>>>>>>> to > > > > >>>>>>>>>>> define > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials will > > > suddenly > > > > >>>>>>>> become > > > > >>>>>>>>>>>> confusing as they all refer to workflows as "DAG"s. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> We are already at risk of scaring the users away with a > > > number > > > > >>>>>> of > > > > >>>>>>>>>>> breaking > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking changes > > > for > > > > >>>>>> the > > > > >>>>>>>>> most > > > > >>>>>>>>>>>> basic things is not something that people are looking for. > > > > >>>>>>>> Attempting > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted as an > > even > > > > >>>>>>>> stronger > > > > >>>>>>>>>>> signal > > > > >>>>>>>>>>>> of project immaturity. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term in the > > > long > > > > >>>>>>>> run. I > > > > >>>>>>>>>>> even > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the DAG > > > > >> definition > > > > >>>>>>>>>>> interface. > > > > >>>>>>>>>>>> We better put our time and efforts in other places in > > > Airflow, > > > > >>>>>> of > > > > >>>>>>>>> which > > > > >>>>>>>>>>>> there are plenty. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Kind regards, > > > > >>>>>>>>>>>> Igor > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak > > > > >>>>>>>>>> <b...@astronomer.io.inva <mailto:b...@astronomer.io.inva>lid > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> Couple of thoughts: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have already faded > > > over > > > > >>>>>>>> time, > > > > >>>>>>>>>>>>> for example there are now several ways to create cyclic > > > > >>>>>> graphs, > > > > >>>>>>>>> e.g. > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these > > properties > > > > >>>>>>>>> vanishing > > > > >>>>>>>>>>>>> even more in the future, so from that perspective I > > support > > > > >>>>>>>>> changing > > > > >>>>>>>>>>>>> âDAG" to a more generic name. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming: > > > > >>>>>>>>>>>>> Dagster: pipeline > > > > >>>>>>>>>>>>> Prefect: flow > > > > >>>>>>>>>>>>> Flyte: workflow > > > > >>>>>>>>>>>>> Temporal: workflow > > > > >>>>>>>>>>>>> Kestra: flow > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I suggest > > > defining > > > > >> a > > > > >>>>>>>>> clear > > > > >>>>>>>>>>>>> path forward. Would we first introduce the deprecation in > > > > >>>>>>>> Airflow > > > > >>>>>>>>> 3, > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4? > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Bas > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <neil4r...@gmail.com > > <mailto:neil4r...@gmail.com>> > > > wrote: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, especially when > > > > >>>>>> most > > > > >>>>>>>>> other > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly. > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing about it > > at > > > > >>>>>>>> all, > > > > >>>>>>>>>>>>>> changing the term though would be fairly confusing to > > most > > > > >>>>>>>> users > > > > >>>>>>>>>> who > > > > >>>>>>>>>>>>>> have been > > > > >>>>>>>>>>>>> using > > > > >>>>>>>>>>>>>> the term for years. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung > > > > >>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto:t...@astronomer.io.inva > > >lid > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term DAG. The > > > only > > > > >>>>>>>>>> problem > > > > >>>>>>>>>>>>> (aside > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding myselfâto stop > > > using > > > > >>>>>>>> the > > > > >>>>>>>>>>>>>>> term) > > > > >>>>>>>>>>>>> is to > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned âworkflowâ is > > > not > > > > >>>>>>>> very > > > > >>>>>>>>>>>>> accurate > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was proposed, but > > > itâs a > > > > >>>>>>>> bit > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and kept > > finding > > > > >>>>>>>> myself > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to clarify âDAG > > > > >>>>>>>>>> definitionâ > > > > >>>>>>>>>>>>>>> (defeating the purpose). > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> TP > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler > > > > >>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > > j_scheff...@gmx.de.inva>LID> > > > > >>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Hi Ryan, > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same > > > observation, > > > > >>>>>>>> had a > > > > >>>>>>>>>>>>>>>> short > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an > > introduction > > > > >>>>>> if > > > > >>>>>>>>>>>>>>> someone > > > > >>>>>>>>>>>>> joins > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes sense. > > > > >>>>>> Especially > > > > >>>>>>>>> when > > > > >>>>>>>>>>>>>>> we > > > > >>>>>>>>>>>>> also > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a reasonable step. > > > > >>>>>>>> Concepts > > > > >>>>>>>>>>>>>>> still > > > > >>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>> stay the same. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding below the desk > > > with > > > > >>>>>>>> you > > > > >>>>>>>>> and > > > > >>>>>>>>>>>>>>>> +1 > > > > >>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>> raising the discussion. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep details of > > > > >>>>>> python > > > > >>>>>>>>> names > > > > >>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>> same because the execution is still a DAG⊠but user > > > facing > > > > >>>>>> it > > > > >>>>>>>>> is a > > > > >>>>>>>>>>>>> workflow. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Jens > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter < > > > > >>>>>>>>>> ryan.hat...@astronomer.io <mailto:ryan.hat...@astronomer.io > > > > > > > >>>>>>>>>>>>> .invalid> > > > > >>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at least for > > > now. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to Airflow > > > users. > > > > >>>>>>>>> Indeed, > > > > >>>>>>>>>>>>>>>>> it > > > > >>>>>>>>>>>>> has > > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians and > > > > >>>>>> software > > > > >>>>>>>>>>>>>>>>> engineers > > > > >>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter. For > > > someone > > > > >>>>>>>> new > > > > >>>>>>>>> to > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, one of > > the > > > > >>>>>>>> first > > > > >>>>>>>>>>>>>>>>> questions they > > > > >>>>>>>>>>>>>>> will > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The answer > > is > > > > >>>>>>>> almost > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You don't > > need > > > to > > > > >>>>>>>>> worry > > > > >>>>>>>>>>>>>>>>> about what > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow." > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a couple > > > > >>>>>>>> important > > > > >>>>>>>>>>>> reasons: > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned above, "DAG" > > > is > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We want > > > Airflow > > > > >>>>>>>> to > > > > >>>>>>>>> be > > > > >>>>>>>>>>>>>>>>> approachable, and > > > > >>>>>>>>>>>>>>> using > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat creates > > > an > > > > >>>>>>>>> initial > > > > >>>>>>>>>>>>>>> barrier to > > > > >>>>>>>>>>>>>>>>> understanding. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow Concepts*: The > > DAG > > > > >>>>>> is > > > > >>>>>>>>> just > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The workflow > > > > >>>>>> includes > > > > >>>>>>>>> its > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other > > parameters, > > > > >>>>>> and > > > > >>>>>>>>> other > > > > >>>>>>>>>>>>>>>>> metadata that > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow homepage > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > > https://airflow.apache.org/>>. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by the > > community > > > to > > > > >>>>>>>>>>>>>>> programmatically > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows. > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" docs page > > > > >>>>>>>>>>>>>>>>> < > > > > >>>>>>>>>> > > > > >> https://airflow.apache.org/docs/apache-airflow/stable/index.html < > > https://airflow.apache.org/docs/apache-airflow/stable/index.html> > > > > >>>>>>>>>>>>>>>>>> , > > > > >>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>> can > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is without > > using > > > > >>>>>>>> "DAG." > > > > >>>>>>>>>>>>>>>>> It's > > > > >>>>>>>>>>>>>>> only in > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is > > introduced > > > > >>>>>> out > > > > >>>>>>>> of > > > > >>>>>>>>>>>>>>>>> nowhere > > > > >>>>>>>>>>>>>>> as a > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it: > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection of tasks > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these > > > > >>>>>> introductions > > > > >>>>>>>> to > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate DAGs; > > it > > > > >>>>>>>>>>> orchestrates > > > > >>>>>>>>>>>> workflows*. > > > > >>>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant to > > almost > > > > >>>>>>>> every > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for the term > > > "DAG" > > > > >>>>>>>> and > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with > > > "workflow". > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For example, instead of... > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> @dag( > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > >>>>>>>>>>>>>>>>> ... > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > > > >>>>>>>>>>>>>>>>> ) > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Users could do... > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> @workflow( > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > >>>>>>>>>>>>>>>>> ... > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > > > >>>>>>>>>>>>>>>>> ) > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >> ------------------------------------------------------------------ > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > dev-unsubscr...@airflow.apache.org> > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail: > > > > >>>>>>>> dev-h...@airflow.apache.org <mailto: > > dev-h...@airflow.apache.org> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>> ------------------------------------------------------------------- > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail: > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > dev-unsubscr...@airflow.apache.org> > > > > >>>>>>>>>>>>>>> For additional commands, e-mail: > > > > >>>>>> dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org > > > > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>> > > > > >>> > > --------------------------------------------------------------------- > > > > >>>>>>>>>>>> To unsubscribe, e-mail: > > dev-unsubscr...@airflow.apache.org <mailto: > > dev-unsubscr...@airflow.apache.org> > > > > >>>>>>>>>>>> For additional commands, e-mail: > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > > dev-unsubscr...@airflow.apache.org> > > > > For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: > > dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
