Here's a PR to use "dag" as a word in the new 3.0 UI: https://github.com/apache/airflow/pull/43325 Let me know if that's the direction we want to go.
On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar <rbish...@amazon.com.invalid> wrote: > I think Brent & Daniel summarized it best, "dag" is synonymous with > workflows in Airflow through the way we talk and explain what Airflow is > all about. Although folks would ask, I don’t ever use the mathematical > definition of DAG. > It will be challenging and possibly confusing for many users making such a > change - I would rather direct that energy to appending the Oxford > definition of "dag" to include a reference to workflows in Airflow. __ > > -- Rajesh > > > > > > > On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > ja...@potiuk.com>> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > > > > From Guido's post: > > > "Around this time the renaming seems to have been renamed". > > > Naming is hard. > > > J. > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev5...@gmail.com <mailto: > droiddev5...@gmail.com>> wrote: > > > > Yes, completely agree with above comments, dag is an Airflow term now > > rather > > than just a "directed acyclic graph". Believe it or not, I've worked with > > work > > folks who've been using dags in Airflow for years now and are pro devs > but > > have trouble remembering the full form of a DAG! For them, dag = Airflow. > > > > workflow, pipeline, flow are surely better terms but will be a major > > behavior > > change for the developer community, so we'll need a solid plan on how to > > introduce it, when we do. > > > > While we discuss this, I'd like to share about the Great Renaming in core > > Python started by Guido back in 2009. We could probably get some > learnings > > from there? Who knows! > > > > Guido's blog (2009): > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > <https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html> > > Follow-up discussion (2024): > > https://discuss.python.org/t/finishing-the-great-renaming/54082 < > https://discuss.python.org/t/finishing-the-great-renaming/54082> > > > > Cheers, > > Omkar > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee...@gmail.com <mailto: > weilee...@gmail.com>> wrote: > > > > > I think we should probably just accept it as an airflow term. At least, > > > that’s how I understood it when I first used Airflow. I feel renaming > it > > at > > > this stage would require considerable effort from maintainers and > > existing > > > users without providing equivalent benefits. > > > > > > Best, > > > Wei > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxiln...@gmail.com > <mailto:kaxiln...@gmail.com>> wrote: > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this discussion > > for > > > > Airflow 4 :) -- but right now it will cause too much disruption > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > > <consta...@astronomer.io.inva <mailto:consta...@astronomer.io.inva>lid> > wrote: > > > > > > > >> In my experience, when you ask those with Airflow experience what a > > dag > > > is, > > > >> they'll start talking about workflow attributes - stuff like dags > > being > > > a > > > >> series of steps or tasks with owners. The structure doesn't come up. > > > >> > > > >> Echo-ing others, at this point, my vote is to embrace the name and > > > >> de-emphasize the mathematical structure aspect. > > > >> > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>lid> > > > >> wrote: > > > >> > > > >>> It's an interesting discussion and I remember struggling with this > > > when I > > > >>> started working with Airflow. > > > >>> But, I also agree with the viewpoint of it being an established > > concept > > > >> now > > > >>> regardless of the origin. > > > >>> > > > >>> I am personally leaning towards the perspective best expressed by > > > Daniel > > > >>> Standish and Brent of using Dag as a word, rather than the computer > > > >> science > > > >>> concept. > > > >>> > > > >>> Best regards, > > > >>> Vikram > > > >>> > > > >>> > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > >> <oniko...@amazon.com.inva <mailto:oniko...@amazon.com.inva>lid > > > >>>> > > > >>> wrote: > > > >>> > > > >>>> I agree with the general sentiment of: You're right Ryan, DAG > isn't > > > >> great > > > >>>> and I'd rather workflow, but changing it will cause much more > > wreckage > > > >>> than > > > >>>> it solves. > > > >>>> > > > >>>> Also agree with the idea to just move away from defining DAG. I > > think > > > >>>> we've been naturally doing that as a community for a while now > > anyway, > > > >> so > > > >>>> that feels like a natural step. > > > >>>> > > > >>>> Cheers, > > > >>>> Niko > > > >>>> > > > >>>> ________________________________ > > > >>>> From: Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > >>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org> > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end > > > >> users > > > >>>> > > > >>>> CAUTION: This email originated from outside of the organization. > Do > > > not > > > >>>> click links or open attachments unless you can confirm the sender > > and > > > >>> know > > > >>>> the content is safe. > > > >>>> > > > >>>> > > > >>>> > > > >>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > >> externe. > > > >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne > > > >>> pouvez > > > >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain > > > >>> que > > > >>>> le contenu ne présente aucun risque. > > > >>>> > > > >>>> > > > >>>> > > > >>>> Best argument in favour of keeping “dags” as a term — getting to > > > re-use > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg < > https://i.imgflip.com/1xhtwh.jpg> > > > >>>> > > > >>>> In all seriousness: I don’t mind either way, both sides have good > > > >> reasons > > > >>>> presented. > > > >>>> > > > >>>> -a > > > >>>> > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > >>>> <daniel.stand...@astronomer.io.inva <mailto: > daniel.stand...@astronomer.io.inva>LID> wrote: > > > >>>>> > > > >>>>> Yeah just say, when asked where the name comes from, "well, no > one > > > >>>> actually > > > >>>>> knows but..." and then make something up. > > > >>>>> > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com>> > > > >> wrote: > > > >>>>> > > > >>>>>> Just to clarify - "directed acyclic graph" is the > tongue-twister, > > > >>>>>> > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <ja...@potiuk.com > <mailto:ja...@potiuk.com>> > > > >>> wrote: > > > >>>>>> > > > >>>>>>> I like what both Daniel and Brent wrote. I would very much want > > to > > > >> be > > > >>>>>> able > > > >>>>>>> to say just "dag" without explaining it further. > > > >>>>>>> > > > >>>>>>> For me every time I explain "DAG" at a talk it's a > > tongue-twister, > > > >>> and > > > >>>> I > > > >>>>>>> almost stutter on trying to recall how to pronounce it > properly. > > > >>>>>>> > > > >>>>>>> J. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > > >>>>>> <br...@astronomer.io.inva <mailto:br...@astronomer.io.inva>lid> > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> I remember we explored renaming "DAG" when starting on AIP-38 > to > > > >>>>>> modernize > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more descriptive of > > what > > > >>> one > > > >>>>>> is > > > >>>>>>>> actually doing while Directed Acyclic Graph is an > implementation > > > >>>> detail. > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG" has > become > > > >>> "dag" > > > >>>>>> , a > > > >>>>>>>> word in its own right. > > > >>>>>>>> > > > >>>>>>>> Examples for "dag" are abound in community discussion, Airflow > > > >>> Summit > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace "dag". > A > > > >> user > > > >>>>>> just > > > >>>>>>>> needs to learn one new word vs the technical concept behind > that > > > >>>> word. I > > > >>>>>>>> think that is much less effort than refactoring so much code, > > > >>>>>>>> documentation, blog posts, stack overflow questions, etc. > > > >>>>>>>> > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish > > > >>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > daniel.stand...@astronomer.io.inva>lid> wrote: > > > >>>>>>>> > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain for > > > >>>> questionable > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I agree the > > > >>>>>>>> association > > > >>>>>>>>> with "directed acyclic graph" is not helpful. > > > >>>>>>>>> > > > >>>>>>>>> And along those lines, I offer here some less invasive > > > >> mitigations. > > > >>>>>>>>> > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize the > math > > > >> nerd > > > >>>>>>>> origins > > > >>>>>>>>> of the name. That is to say, in docs / website / etc, *never > > > >>> define* > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph. Always > > > >> define > > > >>>> it > > > >>>>>>>> as a > > > >>>>>>>>> pipeline, collection of tasks, workflow etc. > > > >>>>>>>>> > > > >>>>>>>>> The "directed acyclic graph" part of it is like a historical > > > >>>> footnote, > > > >>>>>>>> and > > > >>>>>>>>> we could make one mention of it somewhere hidden. > > > >>>>>>>>> > > > >>>>>>>>> We could also start using lowercase in the docs in general > e.g. > > > >>>>>> writing > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc. The > > upper > > > >>> case > > > >>>>>>>> part > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in airlfow is > > just > > > >>> an > > > >>>>>>>>> airflow concept and the association with "DAGs" is not really > > > >>>>>> unhelpful. > > > >>>>>>>>> > > > >>>>>>>>> In other words embrace that "dag" in airflow is its own > thing, > > is > > > >>>>>>>>> *not* strictly > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows about > > > >>> anyway), > > > >>>>>> and > > > >>>>>>>>> tell them what it is in simple terms that normal people > > > >> understand. > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk < > ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > > > >>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> DAG is so embedded into what we do that it will be extremely > > > >>>>>>>> difficult to > > > >>>>>>>>>> get rid of it completely. Also I think it will make a lot of > > > >>>>>> "google" > > > >>>>>>>>>> searches and "stack overflow" searches not finding the right > > > >>>>>> answers. > > > >>>>>>>>> This > > > >>>>>>>>>> is one of the strengths of Airflow - besides the community > and > > > >>> ideas > > > >>>>>>>> that > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples, problems > and > > > >>>>>>>> solutions > > > >>>>>>>>>> you can so easily find (and we have to remember that all the > > AI > > > >>>>>>>> trained > > > >>>>>>>>> on > > > >>>>>>>>>> past data will be also rather poorly matching queries of > > people. > > > >>>>>>>>>> > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch. And if > we > > > >> do > > > >>> - > > > >>>>>> I > > > >>>>>>>>>> would be for using workflow or pipeline instead of `dag` if > > not > > > >>> the > > > >>>>>>>> above > > > >>>>>>>>>> reason, but I think I am here with Igor that it might cause > > more > > > >>>>>>>> problems > > > >>>>>>>>>> than it solves. > > > >>>>>>>>>> > > > >>>>>>>>>> But I am not 100% against - if others will think it's a good > > > >>> idea, I > > > >>>>>>>> am > > > >>>>>>>>> ok > > > >>>>>>>>>> with it. > > > >>>>>>>>>> > > > >>>>>>>>>> J, > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat > > > >>>>>>>>>> <abhishek.bha...@astronomer.io.inva <mailto: > abhishek.bha...@astronomer.io.inva>lid> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to someone > new > > > >> to > > > >>>>>>>>> workflow > > > >>>>>>>>>>> orchestration. But it does also show the nature of being > > > >> acyclic. > > > >>>>>>>> Sure, > > > >>>>>>>>>> as > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it. Still, in > my > > > >>>>>>>> opinion, > > > >>>>>>>>>> there > > > >>>>>>>>>>> is generally no need for cyclic behavior in workflow > > > >>>>>> orchestration. > > > >>>>>>>>> Most > > > >>>>>>>>>>> (*if > > > >>>>>>>>>>> not all*) cases can be in some way can be covered using an > > > >>> acyclic > > > >>>>>>>>> manner > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I would want > > the > > > >>>>>>>>> "acyclic" > > > >>>>>>>>>>> word to stick. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Regards, > > > >>>>>>>>>>> Avi > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > bernd.stroe...@kosakya.de <mailto:bernd.stroe...@kosakya.de>> > > > >>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow Fan; so > many > > > >> new > > > >>>>>>>>> ideas. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced by the > > more > > > >>>>>>>> general > > > >>>>>>>>>>> Term > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri) Net > (AFN) > > > >>>>>> (maybe > > > >>>>>>>>>>> without > > > >>>>>>>>>>>> a direction); > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph Database. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG) might be > > > >>>>>>>> assigned > > > >>>>>>>>> to > > > >>>>>>>>>> an > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a library. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> A running Graph might have a different structure than a > > > >>>>>>>> configuration > > > >>>>>>>>>>>> Graph. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Forget that if you think it's bullshit. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best Regards > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Bernd Ströhle > > > >>>>>>>>>>>> M: +49 171 5357916 > > > >>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > bernd.stroe...@gmail.com> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> -----Original Message----- > > > >>>>>>>>>>>> From: Igor Kholopov <ikholo...@google.com.inva <mailto: > ikholo...@google.com.inva>LID> > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term "DAG" for > end > > > >>>>>> users > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, it is > part > > > >> of > > > >>>>>>>>> Airflow > > > >>>>>>>>>>>> DAG definition interface at so many levels, that any > attempt > > > >> to > > > >>>>>>>>> change > > > >>>>>>>>>> it > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The only > > thing > > > >>>>>>>> that is > > > >>>>>>>>>>> worse > > > >>>>>>>>>>>> than a poorly chosen name in the code is when there are > two > > > >> ways > > > >>>>>>>> to > > > >>>>>>>>>>> define > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials will > > suddenly > > > >>>>>>>> become > > > >>>>>>>>>>>> confusing as they all refer to workflows as "DAG"s. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> We are already at risk of scaring the users away with a > > number > > > >>>>>> of > > > >>>>>>>>>>> breaking > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking changes > > for > > > >>>>>> the > > > >>>>>>>>> most > > > >>>>>>>>>>>> basic things is not something that people are looking for. > > > >>>>>>>> Attempting > > > >>>>>>>>>> to > > > >>>>>>>>>>>> change the fundamental terms will be interpreted as an > even > > > >>>>>>>> stronger > > > >>>>>>>>>>> signal > > > >>>>>>>>>>>> of project immaturity. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term in the > > long > > > >>>>>>>> run. I > > > >>>>>>>>>>> even > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the DAG > > > >> definition > > > >>>>>>>>>>> interface. > > > >>>>>>>>>>>> We better put our time and efforts in other places in > > Airflow, > > > >>>>>> of > > > >>>>>>>>> which > > > >>>>>>>>>>>> there are plenty. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Kind regards, > > > >>>>>>>>>>>> Igor > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak > > > >>>>>>>>>> <b...@astronomer.io.inva <mailto:b...@astronomer.io.inva>lid > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Couple of thoughts: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> 1. The boundaries/properties of “DAG” have already faded > > over > > > >>>>>>>> time, > > > >>>>>>>>>>>>> for example there are now several ways to create cyclic > > > >>>>>> graphs, > > > >>>>>>>>> e.g. > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these > properties > > > >>>>>>>>> vanishing > > > >>>>>>>>>>>>> even more in the future, so from that perspective I > support > > > >>>>>>>>> changing > > > >>>>>>>>>>>>> “DAG" to a more generic name. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming: > > > >>>>>>>>>>>>> Dagster: pipeline > > > >>>>>>>>>>>>> Prefect: flow > > > >>>>>>>>>>>>> Flyte: workflow > > > >>>>>>>>>>>>> Temporal: workflow > > > >>>>>>>>>>>>> Kestra: flow > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> I think “workflow” is the most fitting name. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I suggest > > defining > > > >> a > > > >>>>>>>>> clear > > > >>>>>>>>>>>>> path forward. Would we first introduce the deprecation in > > > >>>>>>>> Airflow > > > >>>>>>>>> 3, > > > >>>>>>>>>>>>> and remove “DAG” in Airflow 4? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Bas > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <neil4r...@gmail.com > <mailto:neil4r...@gmail.com>> > > wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, especially when > > > >>>>>> most > > > >>>>>>>>> other > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly. > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing about it > at > > > >>>>>>>> all, > > > >>>>>>>>>>>>>> changing the term though would be fairly confusing to > most > > > >>>>>>>> users > > > >>>>>>>>>> who > > > >>>>>>>>>>>>>> have been > > > >>>>>>>>>>>>> using > > > >>>>>>>>>>>>>> the term for years. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung > > > >>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto:t...@astronomer.io.inva > >lid > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term DAG. The > > only > > > >>>>>>>>>> problem > > > >>>>>>>>>>>>> (aside > > > >>>>>>>>>>>>>>> from actually telling people—including myself—to stop > > using > > > >>>>>>>> the > > > >>>>>>>>>>>>>>> term) > > > >>>>>>>>>>>>> is to > > > >>>>>>>>>>>>>>> come up with a reasonable alternative. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> I can’t recall who, but someone mentioned “workflow” is > > not > > > >>>>>>>> very > > > >>>>>>>>>>>>> accurate > > > >>>>>>>>>>>>>>> for Airflow. The term “definition” was proposed, but > > it’s a > > > >>>>>>>> bit > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and kept > finding > > > >>>>>>>> myself > > > >>>>>>>>>>>>>>> doubting “what definition?” and wanting to clarify “DAG > > > >>>>>>>>>> definition” > > > >>>>>>>>>>>>>>> (defeating the purpose). > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> TP > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler > > > >>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > j_scheff...@gmx.de.inva>LID> > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Hi Ryan, > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same > > observation, > > > >>>>>>>> had a > > > >>>>>>>>>>>>>>>> short > > > >>>>>>>>>>>>>>> laight because the DAG question is always an > introduction > > > >>>>>> if > > > >>>>>>>>>>>>>>> someone > > > >>>>>>>>>>>>> joins > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes sense. > > > >>>>>> Especially > > > >>>>>>>>> when > > > >>>>>>>>>>>>>>> we > > > >>>>>>>>>>>>> also > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a reasonable step. > > > >>>>>>>> Concepts > > > >>>>>>>>>>>>>>> still > > > >>>>>>>>>>>>> can > > > >>>>>>>>>>>>>>> stay the same. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding below the desk > > with > > > >>>>>>>> you > > > >>>>>>>>> and > > > >>>>>>>>>>>>>>>> +1 > > > >>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>> raising the discussion. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep details of > > > >>>>>> python > > > >>>>>>>>> names > > > >>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>> same because the execution is still a DAG… but user > > facing > > > >>>>>> it > > > >>>>>>>>> is a > > > >>>>>>>>>>>>> workflow. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Jens > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Sent from my Smartphone > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter < > > > >>>>>>>>>> ryan.hat...@astronomer.io <mailto:ryan.hat...@astronomer.io > > > > > >>>>>>>>>>>>> .invalid> > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at least for > > now. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to Airflow > > users. > > > >>>>>>>>> Indeed, > > > >>>>>>>>>>>>>>>>> it > > > >>>>>>>>>>>>> has > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians and > > > >>>>>> software > > > >>>>>>>>>>>>>>>>> engineers > > > >>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter. For > > someone > > > >>>>>>>> new > > > >>>>>>>>> to > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, one of > the > > > >>>>>>>> first > > > >>>>>>>>>>>>>>>>> questions they > > > >>>>>>>>>>>>>>> will > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The answer > is > > > >>>>>>>> almost > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You don't > need > > to > > > >>>>>>>>> worry > > > >>>>>>>>>>>>>>>>> about what > > > >>>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow." > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a couple > > > >>>>>>>> important > > > >>>>>>>>>>>> reasons: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned above, "DAG" > > is > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We want > > Airflow > > > >>>>>>>> to > > > >>>>>>>>> be > > > >>>>>>>>>>>>>>>>> approachable, and > > > >>>>>>>>>>>>>>> using > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat creates > > an > > > >>>>>>>>> initial > > > >>>>>>>>>>>>>>> barrier to > > > >>>>>>>>>>>>>>>>> understanding. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow Concepts*: The > DAG > > > >>>>>> is > > > >>>>>>>>> just > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The workflow > > > >>>>>> includes > > > >>>>>>>>> its > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other > parameters, > > > >>>>>> and > > > >>>>>>>>> other > > > >>>>>>>>>>>>>>>>> metadata that > > > >>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>> DAG component doesn’t account for. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow homepage > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > https://airflow.apache.org/>>. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Apache Airflow® is a platform created by the > community > > to > > > >>>>>>>>>>>>>>> programmatically > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows. > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" docs page > > > >>>>>>>>>>>>>>>>> < > > > >>>>>>>>>> > > > >> https://airflow.apache.org/docs/apache-airflow/stable/index.html < > https://airflow.apache.org/docs/apache-airflow/stable/index.html> > > > >>>>>>>>>>>>>>>>>> , > > > >>>>>>>>>>>>> we > > > >>>>>>>>>>>>>>> can > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is without > using > > > >>>>>>>> "DAG." > > > >>>>>>>>>>>>>>>>> It's > > > >>>>>>>>>>>>>>> only in > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is > introduced > > > >>>>>> out > > > >>>>>>>> of > > > >>>>>>>>>>>>>>>>> nowhere > > > >>>>>>>>>>>>>>> as a > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection of tasks > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these > > > >>>>>> introductions > > > >>>>>>>> to > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate DAGs; > it > > > >>>>>>>>>>> orchestrates > > > >>>>>>>>>>>> workflows*. > > > >>>>>>>>>>>>>>> The > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant to > almost > > > >>>>>>>> every > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for the term > > "DAG" > > > >>>>>>>> and > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with > > "workflow". > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> For example, instead of... > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> @dag( > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > >>>>>>>>>>>>>>>>> ... > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > > >>>>>>>>>>>>>>>>> ) > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Users could do... > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> @workflow( > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > >>>>>>>>>>>>>>>>> ... > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > > >>>>>>>>>>>>>>>>> ) > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> And with that... I will start running away. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>> > > > >> ------------------------------------------------------------------ > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> > > > >>>>>>>>>>>>>>>> For additional commands, e-mail: > > > >>>>>>>> dev-h...@airflow.apache.org <mailto: > dev-h...@airflow.apache.org> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>> > > > >>> ------------------------------------------------------------------- > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail: > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> > > > >>>>>>>>>>>>>>> For additional commands, e-mail: > > > >>>>>> dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org > > > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>> > > > >>> > --------------------------------------------------------------------- > > > >>>>>>>>>>>> To unsubscribe, e-mail: > dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> > > > >>>>>>>>>>>> For additional commands, e-mail: > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > dev-unsubscr...@airflow.apache.org> > > > For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: > dev-h...@airflow.apache.org> > > > > > > > > > > > >
