Just to clarify - "directed acyclic graph" is the tongue-twister,
On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <ja...@potiuk.com> wrote: > I like what both Daniel and Brent wrote. I would very much want to be able > to say just "dag" without explaining it further. > > For me every time I explain "DAG" at a talk it's a tongue-twister, and I > almost stutter on trying to recall how to pronounce it properly. > > J. > > > On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi <br...@astronomer.io.invalid> > wrote: > >> I remember we explored renaming "DAG" when starting on AIP-38 to modernize >> the UI. Both "pipeline" or "workflow" are more descriptive of what one is >> actually doing while Directed Acyclic Graph is an implementation detail. >> But I agree with Daniel Standish, at this point "DAG" has become "dag" , a >> word in its own right. >> >> Examples for "dag" are abound in community discussion, Airflow Summit >> talks, documentation and even in the UI. Let's embrace "dag". A user just >> needs to learn one new word vs the technical concept behind that word. I >> think that is much less effort than refactoring so much code, >> documentation, blog posts, stack overflow questions, etc. >> >> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish >> <daniel.stand...@astronomer.io.invalid> wrote: >> >> > I am skeptical. Seems like introducing a lot of pain for questionable >> > benefit. But, I am def sympathetic to the idea. I agree the >> association >> > with "directed acyclic graph" is not helpful. >> > >> > And along those lines, I offer here some less invasive mitigations. >> > >> > One thing we can do no matter what is to de-emphasize the math nerd >> origins >> > of the name. That is to say, in docs / website / etc, *never define* >> > airflow's "dag" concept as a directed acyclic graph. Always define it >> as a >> > pipeline, collection of tasks, workflow etc. >> > >> > The "directed acyclic graph" part of it is like a historical footnote, >> and >> > we could make one mention of it somewhere hidden. >> > >> > We could also start using lowercase in the docs in general e.g. writing >> > "dag" / "dags" instead of writing "DAG" / "DAGs" etc. The upper case >> part >> > of it makes it look like an acronym; but "dag" in airlfow is just an >> > airflow concept and the association with "DAGs" is not really unhelpful. >> > >> > In other words embrace that "dag" in airflow is its own thing, is >> > *not* strictly >> > speaking a directed acyclic graph (which nobody knows about anyway), and >> > tell them what it is in simple terms that normal people understand. >> > >> > >> > On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> > > DAG is so embedded into what we do that it will be extremely >> difficult to >> > > get rid of it completely. Also I think it will make a lot of "google" >> > > searches and "stack overflow" searches not finding the right answers. >> > This >> > > is one of the strengths of Airflow - besides the community and ideas >> that >> > > Bernd mentioned - is the vast number of examples, problems and >> solutions >> > > you can so easily find (and we have to remember that all the AI >> trained >> > on >> > > past data will be also rather poorly matching queries of people. >> > > >> > > I am not too attached to DAG. I could easily switch. And if we do - I >> > > would be for using workflow or pipeline instead of `dag` if not the >> above >> > > reason, but I think I am here with Igor that it might cause more >> problems >> > > than it solves. >> > > >> > > But I am not 100% against - if others will think it's a good idea, I >> am >> > ok >> > > with it. >> > > >> > > J, >> > > >> > > >> > > On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat >> > > <abhishek.bha...@astronomer.io.invalid> wrote: >> > > >> > > > Agreed that the word DAG makes very less sense to someone new to >> > workflow >> > > > orchestration. But it does also show the nature of being acyclic. >> Sure, >> > > as >> > > > Bas mentioned, there are ways to workaround it. Still, in my >> opinion, >> > > there >> > > > is generally no need for cyclic behavior in workflow orchestration. >> > Most >> > > > (*if >> > > > not all*) cases can be in some way can be covered using an acyclic >> > manner >> > > > with multiple runs. Hence, the idempotency. So I would want the >> > "acyclic" >> > > > word to stick. >> > > > >> > > > Regards, >> > > > Avi >> > > > >> > > > On Tue, Oct 22, 2024 at 12:41 PM <bernd.stroe...@kosakya.de> wrote: >> > > > >> > > > > Brilliant, I am on the way to become an Airflow Fan; so many new >> > ideas. >> > > > > >> > > > > The Term DAG is misleading; it should be replaced by the more >> general >> > > > Term >> > > > > Airflow (Workflow) Graph (AFG) or Airflow (Petri) Net (AFN) (maybe >> > > > without >> > > > > a direction); >> > > > > and ... these Graphs should be stored in a Graph Database. >> > > > > >> > > > > Every Node or Sup-Graph of an Airflow Graph (AFG) might be >> assigned >> > to >> > > an >> > > > > executable (Python-, Rust-, ... ) member of a library. >> > > > > >> > > > > A running Graph might have a different structure than a >> configuration >> > > > > Graph. >> > > > > >> > > > > Forget that if you think it's bullshit. >> > > > > >> > > > > Best Regards >> > > > > >> > > > > Bernd Ströhle >> > > > > M: +49 171 5357916 >> > > > > E: bernd.stroe...@gmail.com >> > > > > >> > > > > >> > > > > -----Original Message----- >> > > > > From: Igor Kholopov <ikholo...@google.com.INVALID> >> > > > > Sent: Tuesday, October 22, 2024 12:02 PM >> > > > > To: dev@airflow.apache.org >> > > > > Subject: Re: Airflow should deprecate the term "DAG" for end users >> > > > > >> > > > > Even though the term "DAG" is clearly suboptimal, it is part of >> > Airflow >> > > > > DAG definition interface at so many levels, that any attempt to >> > change >> > > it >> > > > > will only introduce more chaos, not reduce it. The only thing >> that is >> > > > worse >> > > > > than a poorly chosen name in the code is when there are two ways >> to >> > > > define >> > > > > the same thing. Countless articles and tutorials will suddenly >> become >> > > > > confusing as they all refer to workflows as "DAG"s. >> > > > > >> > > > > We are already at risk of scaring the users away with a number of >> > > > breaking >> > > > > changes in Airflow 3, promising even more breaking changes for the >> > most >> > > > > basic things is not something that people are looking for. >> Attempting >> > > to >> > > > > change the fundamental terms will be interpreted as an even >> stronger >> > > > signal >> > > > > of project immaturity. >> > > > > >> > > > > Given that, I oppose the idea of changing the term in the long >> run. I >> > > > even >> > > > > stricter oppose the idea of deprecating it in the DAG definition >> > > > interface. >> > > > > We better put our time and efforts in other places in Airflow, of >> > which >> > > > > there are plenty. >> > > > > >> > > > > Kind regards, >> > > > > Igor >> > > > > >> > > > > On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak >> > > <b...@astronomer.io.invalid >> > > > > >> > > > > wrote: >> > > > > >> > > > > > Couple of thoughts: >> > > > > > >> > > > > > 1. The boundaries/properties of “DAG” have already faded over >> time, >> > > > > > for example there are now several ways to create cyclic graphs, >> > e.g. >> > > > > > using the @continuous schedule. I imagine these properties >> > vanishing >> > > > > > even more in the future, so from that perspective I support >> > changing >> > > > > > “DAG" to a more generic name. >> > > > > > >> > > > > > 2. How other orchestration frameworks do naming: >> > > > > > Dagster: pipeline >> > > > > > Prefect: flow >> > > > > > Flyte: workflow >> > > > > > Temporal: workflow >> > > > > > Kestra: flow >> > > > > > >> > > > > > I think “workflow” is the most fitting name. >> > > > > > >> > > > > > 3. Given the large impact of this change, I suggest defining a >> > clear >> > > > > > path forward. Would we first introduce the deprecation in >> Airflow >> > 3, >> > > > > > and remove “DAG” in Airflow 4? >> > > > > > >> > > > > > Bas >> > > > > > >> > > > > > > On 22 Oct 2024, at 09:22, Neil <neil4r...@gmail.com> wrote: >> > > > > > > >> > > > > > > I don't see a problem with the term DAG, especially when most >> > other >> > > > > > > platforms embrace the term wholeheartedly. >> > > > > > > I don't see anything intimidating or confusing about it at >> all, >> > > > > > > changing the term though would be fairly confusing to most >> users >> > > who >> > > > > > > have been >> > > > > > using >> > > > > > > the term for years. >> > > > > > > >> > > > > > > On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung >> > > > > > > <t...@astronomer.io.invalid >> > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > >> I totally agree with doing away with the term DAG. The only >> > > problem >> > > > > > (aside >> > > > > > >> from actually telling people—including myself—to stop using >> the >> > > > > > >> term) >> > > > > > is to >> > > > > > >> come up with a reasonable alternative. >> > > > > > >> >> > > > > > >> I can’t recall who, but someone mentioned “workflow” is not >> very >> > > > > > accurate >> > > > > > >> for Airflow. The term “definition” was proposed, but it’s a >> bit >> > > > > > >> broad; I tried to use it in a few places and kept finding >> myself >> > > > > > >> doubting “what definition?” and wanting to clarify “DAG >> > > definition” >> > > > > > >> (defeating the purpose). >> > > > > > >> >> > > > > > >> TP >> > > > > > >> >> > > > > > >> >> > > > > > >>> On 22 Oct 2024, at 13:07, Jens Scheffler >> > > > > > >>> <j_scheff...@gmx.de.INVALID> >> > > > > > >> wrote: >> > > > > > >>> >> > > > > > >>> Hi Ryan, >> > > > > > >>> >> > > > > > >>> Thanks for posting. I share the exactly same observation, >> had a >> > > > > > >>> short >> > > > > > >> laight because the DAG question is always an introduction if >> > > > > > >> someone >> > > > > > joins >> > > > > > >> the party. I think a global renaming makes sense. Especially >> > when >> > > > > > >> we >> > > > > > also >> > > > > > >> rename Dataset to Asset this is also a reasonable step. >> Concepts >> > > > > > >> still >> > > > > > can >> > > > > > >> stay the same. >> > > > > > >>> >> > > > > > >>> So I hope I don‘t need to join hiding below the desk with >> you >> > and >> > > > > > >>> +1 >> > > > > > for >> > > > > > >> raising the discussion. >> > > > > > >>> >> > > > > > >>> Technically we can still think if we keep details of python >> > names >> > > > > > >>> the >> > > > > > >> same because the execution is still a DAG… but user facing it >> > is a >> > > > > > workflow. >> > > > > > >>> >> > > > > > >>> Jens >> > > > > > >>> >> > > > > > >>> Sent from my Smartphone >> > > > > > >>> >> > > > > > >>>> On 21. Oct 2024, at 23:56, Ryan Hatter < >> > > ryan.hat...@astronomer.io >> > > > > > .invalid> >> > > > > > >> wrote: >> > > > > > >>>> >> > > > > > >>>> Everyone please sheathe your swords... at least for now. >> > > > > > >>>> >> > > > > > >>>> The term "DAG" has very little meaning to Airflow users. >> > Indeed, >> > > > > > >>>> it >> > > > > > has >> > > > > > >>>> little meaning outside of some mathematicians and software >> > > > > > >>>> engineers >> > > > > > for >> > > > > > >>>> whom the properties of a DAG actually matter. For someone >> new >> > to >> > > > > > >>>> data engineering or workflow orchestration, one of the >> first >> > > > > > >>>> questions they >> > > > > > >> will >> > > > > > >>>> likely have is, "what on earth is a DAG?" The answer is >> almost >> > > > > > >>>> always, "It's a directed acyclic graph. You don't need to >> > worry >> > > > > > >>>> about what >> > > > > > that >> > > > > > >>>> means; it's just a term for your workflow." >> > > > > > >>>> >> > > > > > >>>> The term "DAG" is problematic for at least a couple >> important >> > > > > reasons: >> > > > > > >>>> >> > > > > > >>>> *Complexity for New Users*: As mentioned above, "DAG" is >> > > > > > >>>> unnecessarily intimidating and confusing. We want Airflow >> to >> > be >> > > > > > >>>> approachable, and >> > > > > > >> using >> > > > > > >>>> technical jargon like "DAG" right off the bat creates an >> > initial >> > > > > > >> barrier to >> > > > > > >>>> understanding. >> > > > > > >>>> >> > > > > > >>>> *Disconnect Between DAG and Workflow Concepts*: The DAG is >> > just >> > > > > > >>>> one component of an Airflow workflow. The workflow includes >> > its >> > > > > > >>>> schedule, retries, timeouts, a dozen other parameters, and >> > other >> > > > > > >>>> metadata that >> > > > > > the >> > > > > > >>>> DAG component doesn’t account for. >> > > > > > >>>> >> > > > > > >>>> Consider the following from the Airflow homepage >> > > > > > >>>> <https://airflow.apache.org/>. >> > > > > > >>>> >> > > > > > >>>> Apache Airflow® is a platform created by the community to >> > > > > > >> programmatically >> > > > > > >>>> author, schedule and monitor workflows. >> > > > > > >>>> Then, if we look at the "What is Airflow?" docs page >> > > > > > >>>> < >> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html >> > > > > > >>>> >, >> > > > > > we >> > > > > > >> can >> > > > > > >>>> see that the docs explain what Airflow is without using >> "DAG." >> > > > > > >>>> It's >> > > > > > >> only in >> > > > > > >>>> the *workflow* Python code that the term is introduced out >> of >> > > > > > >>>> nowhere >> > > > > > >> as a >> > > > > > >>>> comment that awkwardly tries to explain it: >> > > > > > >>>> >> > > > > > >>>> # A DAG represents a workflow, a collection of tasks >> > > > > > >>>> >> > > > > > >>>> It makes sense to not refer to DAGs in these introductions >> to >> > > > > > >>>> Airflow, because *Airflow doesn't orchestrate DAGs; it >> > > > orchestrates >> > > > > workflows*. >> > > > > > >> The >> > > > > > >>>> DAG is the model that, for reasons irrelevant to almost >> every >> > > > > > >>>> user, workflows must adhere to. >> > > > > > >>>> >> > > > > > >>>> So, I propose at least adding an alias for the term "DAG" >> and >> > > > > > >>>> updating documentation to replace "DAG" with "workflow". >> > > > > > >>>> >> > > > > > >>>> For example, instead of... >> > > > > > >>>> >> > > > > > >>>> @dag( >> > > > > > >>>> schedule="@daily", >> > > > > > >>>> ... >> > > > > > >>>> dagrun_timeout=timedelta(hours=1) >> > > > > > >>>> ) >> > > > > > >>>> >> > > > > > >>>> Users could do... >> > > > > > >>>> >> > > > > > >>>> @workflow( >> > > > > > >>>> schedule="@daily", >> > > > > > >>>> ... >> > > > > > >>>> run_timeout=timedelta(hours=1) >> > > > > > >>>> ) >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> And with that... I will start running away. >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> >> > > ------------------------------------------------------------------ >> > > > > > >>> --- To unsubscribe, e-mail: >> dev-unsubscr...@airflow.apache.org >> > > > > > >>> For additional commands, e-mail: >> dev-h...@airflow.apache.org >> > > > > > >>> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > ------------------------------------------------------------------- >> > > > > > >> -- To unsubscribe, e-mail: >> dev-unsubscr...@airflow.apache.org >> > > > > > >> For additional commands, e-mail: dev-h...@airflow.apache.org >> > > > > > >> >> > > > > > >> >> > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> --------------------------------------------------------------------- >> > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > > > > For additional commands, e-mail: dev-h...@airflow.apache.org >> > > > > >> > > > > >> > > > >> > > >> > >> >
