Sounds like another +1000 files big PR is coming :) ? Any volunteers to make it ? It's fun.
On Mon, Feb 17, 2025 at 6:52 PM Omkar P <droiddev5...@gmail.com> wrote: > +1 for ruff rules :) > > Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow > docs, in line with the new UI changes and to make the renaming > consistent across user-facing pages. > > Regards, > Omkar > > On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > +1 :) . Maybe we could add a ruff rule for that :) > > > > On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> wrote: > > > > > It seems that our current conclusion is to use "dag" or "Dag" instead > of > > > "DAG" whenever possible. Should we replace all "DAG" in the codebase > with > > > "dag" or "Dag"? If it's too late for that (which it might be 🤔), > should > > we > > > at least avoid introducing new "DAG" in the following PRs? > > > > > > Best, > > > Wei > > > > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > > > Here's a PR to use "dag" as a word in the new 3.0 UI: > > > > https://github.com/apache/airflow/pull/43325 > > > > Let me know if that's the direction we want to go. > > > > > > > > On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > > > > <rbish...@amazon.com.invalid> wrote: > > > > > > > > > I think Brent & Daniel summarized it best, "dag" is synonymous with > > > > > workflows in Airflow through the way we talk and explain what > Airflow > > > is > > > > > all about. Although folks would ask, I don’t ever use the > > mathematical > > > > > definition of DAG. > > > > > It will be challenging and possibly confusing for many users making > > > such a > > > > > change - I would rather direct that energy to appending the Oxford > > > > > definition of "dag" to include a reference to workflows in Airflow. > > __ > > > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com <mailto: > > > > > ja...@potiuk.com>> wrote: > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the organization. Do > > not > > > > > click links or open attachments unless you can confirm the sender > and > > > know > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne > > > pouvez > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain > > > que > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From Guido's post: > > > > > > > > > > > > > > > "Around this time the renaming seems to have been renamed". > > > > > > > > > > > > > > > Naming is hard. > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev5...@gmail.com > > > <mailto: > > > > > droiddev5...@gmail.com>> wrote: > > > > > > > > > > > > > > > > Yes, completely agree with above comments, dag is an Airflow term > > now > > > > > > rather > > > > > > than just a "directed acyclic graph". Believe it or not, I've > > worked > > > with > > > > > > work > > > > > > folks who've been using dags in Airflow for years now and are pro > > > devs > > > > > but > > > > > > have trouble remembering the full form of a DAG! For them, dag = > > > Airflow. > > > > > > > > > > > > workflow, pipeline, flow are surely better terms but will be a > > major > > > > > > behavior > > > > > > change for the developer community, so we'll need a solid plan on > > > how to > > > > > > introduce it, when we do. > > > > > > > > > > > > While we discuss this, I'd like to share about the Great Renaming > > in > > > core > > > > > > Python started by Guido back in 2009. We could probably get some > > > > > learnings > > > > > > from there? Who knows! > > > > > > > > > > > > Guido's blog (2009): > > > > > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > < > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > > > > > Follow-up discussion (2024): > > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > < > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082> > > > > > > > > > > > > Cheers, > > > > > > Omkar > > > > > > > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee...@gmail.com > > > <mailto: > > > > > weilee...@gmail.com>> wrote: > > > > > > > > > > > > > I think we should probably just accept it as an airflow term. > At > > > least, > > > > > > > that’s how I understood it when I first used Airflow. I feel > > > renaming > > > > > it > > > > > > at > > > > > > > this stage would require considerable effort from maintainers > and > > > > > > existing > > > > > > > users without providing equivalent benefits. > > > > > > > > > > > > > > Best, > > > > > > > Wei > > > > > > > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxiln...@gmail.com > > > > > <mailto:kaxiln...@gmail.com>> wrote: > > > > > > > > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this > > > discussion > > > > > > for > > > > > > > > Airflow 4 :) -- but right now it will cause too much > disruption > > > > > > > > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > > > > > > <consta...@astronomer.io.inva <mailto: > > > consta...@astronomer.io.inva>lid> > > > > > wrote: > > > > > > > > > > > > > > > >> In my experience, when you ask those with Airflow experience > > > what a > > > > > > dag > > > > > > > is, > > > > > > > >> they'll start talking about workflow attributes - stuff like > > > dags > > > > > > being > > > > > > > a > > > > > > > >> series of steps or tasks with owners. The structure doesn't > > > come up. > > > > > > > >> > > > > > > > >> Echo-ing others, at this point, my vote is to embrace the > name > > > and > > > > > > > >> de-emphasize the mathematical structure aspect. > > > > > > > >> > > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > > > > > <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva > > >lid> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >>> It's an interesting discussion and I remember struggling > with > > > this > > > > > > > when I > > > > > > > >>> started working with Airflow. > > > > > > > >>> But, I also agree with the viewpoint of it being an > > established > > > > > > concept > > > > > > > >> now > > > > > > > >>> regardless of the origin. > > > > > > > >>> > > > > > > > >>> I am personally leaning towards the perspective best > > expressed > > > by > > > > > > > Daniel > > > > > > > >>> Standish and Brent of using Dag as a word, rather than the > > > computer > > > > > > > >> science > > > > > > > >>> concept. > > > > > > > >>> > > > > > > > >>> Best regards, > > > > > > > >>> Vikram > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > > > > > >> <oniko...@amazon.com.inva <mailto:oniko...@amazon.com.inva > > >lid > > > > > > > >>>> > > > > > > > >>> wrote: > > > > > > > >>> > > > > > > > >>>> I agree with the general sentiment of: You're right Ryan, > > DAG > > > > > isn't > > > > > > > >> great > > > > > > > >>>> and I'd rather workflow, but changing it will cause much > > more > > > > > > wreckage > > > > > > > >>> than > > > > > > > >>>> it solves. > > > > > > > >>>> > > > > > > > >>>> Also agree with the idea to just move away from defining > > DAG. > > > I > > > > > > think > > > > > > > >>>> we've been naturally doing that as a community for a while > > now > > > > > > anyway, > > > > > > > >> so > > > > > > > >>>> that feels like a natural step. > > > > > > > >>>> > > > > > > > >>>> Cheers, > > > > > > > >>>> Niko > > > > > > > >>>> > > > > > > > >>>> ________________________________ > > > > > > > >>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: > > > a...@apache.org>> > > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > > > > > >>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org > > > > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG" > > > for end > > > > > > > >> users > > > > > > > >>>> > > > > > > > >>>> CAUTION: This email originated from outside of the > > > organization. > > > > > Do > > > > > > > not > > > > > > > >>>> click links or open attachments unless you can confirm the > > > sender > > > > > > and > > > > > > > >>> know > > > > > > > >>>> the content is safe. > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> AVERTISSEMENT: Ce courrier électronique provient d’un > > > expéditeur > > > > > > > >> externe. > > > > > > > >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > si > > > vous > > > > > ne > > > > > > > >>> pouvez > > > > > > > >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes > > pas > > > > > > certain > > > > > > > >>> que > > > > > > > >>>> le contenu ne présente aucun risque. > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> Best argument in favour of keeping “dags” as a term — > > getting > > > to > > > > > > > re-use > > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > > > > https://i.imgflip.com/1xhtwh.jpg> > > > > > > > >>>> > > > > > > > >>>> In all seriousness: I don’t mind either way, both sides > have > > > good > > > > > > > >> reasons > > > > > > > >>>> presented. > > > > > > > >>>> > > > > > > > >>>> -a > > > > > > > >>>> > > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > > > > > >>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > > daniel.stand...@astronomer.io.inva>LID> wrote: > > > > > > > >>>>> > > > > > > > >>>>> Yeah just say, when asked where the name comes from, > "well, > > > no > > > > > one > > > > > > > >>>> actually > > > > > > > >>>>> knows but..." and then make something up. > > > > > > > >>>>> > > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < > > > ja...@potiuk.com > > > > > <mailto:ja...@potiuk.com>> > > > > > > > >> wrote: > > > > > > > >>>>> > > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the > > > > > tongue-twister, > > > > > > > >>>>>> > > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < > > > ja...@potiuk.com > > > > > <mailto:ja...@potiuk.com>> > > > > > > > >>> wrote: > > > > > > > >>>>>> > > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very > > much > > > want > > > > > > to > > > > > > > >> be > > > > > > > >>>>>> able > > > > > > > >>>>>>> to say just "dag" without explaining it further. > > > > > > > >>>>>>> > > > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a > > > > > > tongue-twister, > > > > > > > >>> and > > > > > > > >>>> I > > > > > > > >>>>>>> almost stutter on trying to recall how to pronounce it > > > > > properly. > > > > > > > >>>>>>> > > > > > > > >>>>>>> J. > > > > > > > >>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > > > > > > >>>>>> <br...@astronomer.io.inva <mailto: > > br...@astronomer.io.inva > > > >lid> > > > > > > > >>>>>>> wrote: > > > > > > > >>>>>>> > > > > > > > >>>>>>>> I remember we explored renaming "DAG" when starting on > > > AIP-38 > > > > > to > > > > > > > >>>>>> modernize > > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more > > > descriptive of > > > > > > what > > > > > > > >>> one > > > > > > > >>>>>> is > > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an > > > > > implementation > > > > > > > >>>> detail. > > > > > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG" > > has > > > > > become > > > > > > > >>> "dag" > > > > > > > >>>>>> , a > > > > > > > >>>>>>>> word in its own right. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> Examples for "dag" are abound in community discussion, > > > Airflow > > > > > > > >>> Summit > > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace > > > "dag". > > > > > A > > > > > > > >> user > > > > > > > >>>>>> just > > > > > > > >>>>>>>> needs to learn one new word vs the technical concept > > > behind > > > > > that > > > > > > > >>>> word. I > > > > > > > >>>>>>>> think that is much less effort than refactoring so > much > > > code, > > > > > > > >>>>>>>> documentation, blog posts, stack overflow questions, > > etc. > > > > > > > >>>>>>>> > > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish > > > > > > > >>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > > daniel.stand...@astronomer.io.inva>lid> wrote: > > > > > > > >>>>>>>> > > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain > > for > > > > > > > >>>> questionable > > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I > agree > > > the > > > > > > > >>>>>>>> association > > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> And along those lines, I offer here some less > invasive > > > > > > > >> mitigations. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize > > the > > > > > math > > > > > > > >> nerd > > > > > > > >>>>>>>> origins > > > > > > > >>>>>>>>> of the name. That is to say, in docs / website / etc, > > > *never > > > > > > > >>> define* > > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph. > > > Always > > > > > > > >> define > > > > > > > >>>> it > > > > > > > >>>>>>>> as a > > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a > > > historical > > > > > > > >>>> footnote, > > > > > > > >>>>>>>> and > > > > > > > >>>>>>>>> we could make one mention of it somewhere hidden. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> We could also start using lowercase in the docs in > > > general > > > > > e.g. > > > > > > > >>>>>> writing > > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc. > > The > > > > > > upper > > > > > > > >>> case > > > > > > > >>>>>>>> part > > > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in > > > airlfow is > > > > > > just > > > > > > > >>> an > > > > > > > >>>>>>>>> airflow concept and the association with "DAGs" is > not > > > really > > > > > > > >>>>>> unhelpful. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its > own > > > > > thing, > > > > > > is > > > > > > > >>>>>>>>> *not* strictly > > > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows > > > about > > > > > > > >>> anyway), > > > > > > > >>>>>> and > > > > > > > >>>>>>>>> tell them what it is in simple terms that normal > people > > > > > > > >> understand. > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk < > > > > > ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > > > > > > > > > > > >>>>>> wrote: > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be > > > extremely > > > > > > > >>>>>>>> difficult to > > > > > > > >>>>>>>>>> get rid of it completely. Also I think it will make > a > > > lot of > > > > > > > >>>>>> "google" > > > > > > > >>>>>>>>>> searches and "stack overflow" searches not finding > the > > > right > > > > > > > >>>>>> answers. > > > > > > > >>>>>>>>> This > > > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the > > > community > > > > > and > > > > > > > >>> ideas > > > > > > > >>>>>>>> that > > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples, > > > problems > > > > > and > > > > > > > >>>>>>>> solutions > > > > > > > >>>>>>>>>> you can so easily find (and we have to remember that > > > all the > > > > > > AI > > > > > > > >>>>>>>> trained > > > > > > > >>>>>>>>> on > > > > > > > >>>>>>>>>> past data will be also rather poorly matching > queries > > of > > > > > > people. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch. > > > And if > > > > > we > > > > > > > >> do > > > > > > > >>> - > > > > > > > >>>>>> I > > > > > > > >>>>>>>>>> would be for using workflow or pipeline instead of > > > `dag` if > > > > > > not > > > > > > > >>> the > > > > > > > >>>>>>>> above > > > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it > might > > > cause > > > > > > more > > > > > > > >>>>>>>> problems > > > > > > > >>>>>>>>>> than it solves. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> But I am not 100% against - if others will think > it's > > a > > > good > > > > > > > >>> idea, I > > > > > > > >>>>>>>> am > > > > > > > >>>>>>>>> ok > > > > > > > >>>>>>>>>> with it. > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> J, > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat > > > > > > > >>>>>>>>>> <abhishek.bha...@astronomer.io.inva <mailto: > > > > > abhishek.bha...@astronomer.io.inva>lid> wrote: > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to > > > someone > > > > > new > > > > > > > >> to > > > > > > > >>>>>>>>> workflow > > > > > > > >>>>>>>>>>> orchestration. But it does also show the nature of > > > being > > > > > > > >> acyclic. > > > > > > > >>>>>>>> Sure, > > > > > > > >>>>>>>>>> as > > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it. > > Still, > > > in > > > > > my > > > > > > > >>>>>>>> opinion, > > > > > > > >>>>>>>>>> there > > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in > workflow > > > > > > > >>>>>> orchestration. > > > > > > > >>>>>>>>> Most > > > > > > > >>>>>>>>>>> (*if > > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered > > > using an > > > > > > > >>> acyclic > > > > > > > >>>>>>>>> manner > > > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I > > would > > > want > > > > > > the > > > > > > > >>>>>>>>> "acyclic" > > > > > > > >>>>>>>>>>> word to stick. > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> Regards, > > > > > > > >>>>>>>>>>> Avi > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > > > > bernd.stroe...@kosakya.de <mailto:bernd.stroe...@kosakya.de>> > > > > > > > >>>>>> wrote: > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow > Fan; > > > so > > > > > many > > > > > > > >> new > > > > > > > >>>>>>>>> ideas. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced > by > > > the > > > > > > more > > > > > > > >>>>>>>> general > > > > > > > >>>>>>>>>>> Term > > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri) > > Net > > > > > (AFN) > > > > > > > >>>>>> (maybe > > > > > > > >>>>>>>>>>> without > > > > > > > >>>>>>>>>>>> a direction); > > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph > > > Database. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG) > > > might be > > > > > > > >>>>>>>> assigned > > > > > > > >>>>>>>>> to > > > > > > > >>>>>>>>>> an > > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a > > library. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> A running Graph might have a different structure > > than > > > a > > > > > > > >>>>>>>> configuration > > > > > > > >>>>>>>>>>>> Graph. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Best Regards > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Bernd Ströhle > > > > > > > >>>>>>>>>>>> M: +49 171 5357916 > > > > > > > >>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > > > > bernd.stroe...@gmail.com> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> -----Original Message----- > > > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholo...@google.com.inva > > > <mailto: > > > > > ikholo...@google.com.inva>LID> > > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > dev@airflow.apache.org > > > > > > > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term > "DAG" > > > for > > > > > end > > > > > > > >>>>>> users > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, > it > > > is > > > > > part > > > > > > > >> of > > > > > > > >>>>>>>>> Airflow > > > > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that > any > > > > > attempt > > > > > > > >> to > > > > > > > >>>>>>>>> change > > > > > > > >>>>>>>>>> it > > > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The > > > only > > > > > > thing > > > > > > > >>>>>>>> that is > > > > > > > >>>>>>>>>>> worse > > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when > there > > > are > > > > > two > > > > > > > >> ways > > > > > > > >>>>>>>> to > > > > > > > >>>>>>>>>>> define > > > > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials > > will > > > > > > suddenly > > > > > > > >>>>>>>> become > > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as > "DAG"s. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> We are already at risk of scaring the users away > > with > > > a > > > > > > number > > > > > > > >>>>>> of > > > > > > > >>>>>>>>>>> breaking > > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking > > > changes > > > > > > for > > > > > > > >>>>>> the > > > > > > > >>>>>>>>> most > > > > > > > >>>>>>>>>>>> basic things is not something that people are > > looking > > > for. > > > > > > > >>>>>>>> Attempting > > > > > > > >>>>>>>>>> to > > > > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted > as > > an > > > > > even > > > > > > > >>>>>>>> stronger > > > > > > > >>>>>>>>>>> signal > > > > > > > >>>>>>>>>>>> of project immaturity. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term > > in > > > the > > > > > > long > > > > > > > >>>>>>>> run. I > > > > > > > >>>>>>>>>>> even > > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the > > DAG > > > > > > > >> definition > > > > > > > >>>>>>>>>>> interface. > > > > > > > >>>>>>>>>>>> We better put our time and efforts in other places > > in > > > > > > Airflow, > > > > > > > >>>>>> of > > > > > > > >>>>>>>>> which > > > > > > > >>>>>>>>>>>> there are plenty. > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> Kind regards, > > > > > > > >>>>>>>>>>>> Igor > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak > > > > > > > >>>>>>>>>> <b...@astronomer.io.inva <mailto: > > b...@astronomer.io.inva > > > >lid > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> Couple of thoughts: > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of “DAG” have > already > > > faded > > > > > > over > > > > > > > >>>>>>>> time, > > > > > > > >>>>>>>>>>>>> for example there are now several ways to create > > > cyclic > > > > > > > >>>>>> graphs, > > > > > > > >>>>>>>>> e.g. > > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these > > > > > properties > > > > > > > >>>>>>>>> vanishing > > > > > > > >>>>>>>>>>>>> even more in the future, so from that > perspective I > > > > > support > > > > > > > >>>>>>>>> changing > > > > > > > >>>>>>>>>>>>> “DAG" to a more generic name. > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming: > > > > > > > >>>>>>>>>>>>> Dagster: pipeline > > > > > > > >>>>>>>>>>>>> Prefect: flow > > > > > > > >>>>>>>>>>>>> Flyte: workflow > > > > > > > >>>>>>>>>>>>> Temporal: workflow > > > > > > > >>>>>>>>>>>>> Kestra: flow > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> I think “workflow” is the most fitting name. > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I > suggest > > > > > > defining > > > > > > > >> a > > > > > > > >>>>>>>>> clear > > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the > > > deprecation in > > > > > > > >>>>>>>> Airflow > > > > > > > >>>>>>>>> 3, > > > > > > > >>>>>>>>>>>>> and remove “DAG” in Airflow 4? > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> Bas > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < > > neil4r...@gmail.com > > > > > <mailto:neil4r...@gmail.com>> > > > > > > wrote: > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, > > especially > > > when > > > > > > > >>>>>> most > > > > > > > >>>>>>>>> other > > > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly. > > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing > > > about it > > > > > at > > > > > > > >>>>>>>> all, > > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly > confusing > > > to > > > > > most > > > > > > > >>>>>>>> users > > > > > > > >>>>>>>>>> who > > > > > > > >>>>>>>>>>>>>> have been > > > > > > > >>>>>>>>>>>>> using > > > > > > > >>>>>>>>>>>>>> the term for years. > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung > > > > > > > >>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: > > > t...@astronomer.io.inva > > > > > >lid > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term > > DAG. > > > The > > > > > > only > > > > > > > >>>>>>>>>> problem > > > > > > > >>>>>>>>>>>>> (aside > > > > > > > >>>>>>>>>>>>>>> from actually telling people—including > myself—to > > > stop > > > > > > using > > > > > > > >>>>>>>> the > > > > > > > >>>>>>>>>>>>>>> term) > > > > > > > >>>>>>>>>>>>> is to > > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative. > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> I can’t recall who, but someone mentioned > > > “workflow” is > > > > > > not > > > > > > > >>>>>>>> very > > > > > > > >>>>>>>>>>>>> accurate > > > > > > > >>>>>>>>>>>>>>> for Airflow. The term “definition” was > proposed, > > > but > > > > > > it’s a > > > > > > > >>>>>>>> bit > > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and > kept > > > > > finding > > > > > > > >>>>>>>> myself > > > > > > > >>>>>>>>>>>>>>> doubting “what definition?” and wanting to > > clarify > > > “DAG > > > > > > > >>>>>>>>>> definition” > > > > > > > >>>>>>>>>>>>>>> (defeating the purpose). > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> TP > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler > > > > > > > >>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > > > > > j_scheff...@gmx.de.inva>LID> > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> Hi Ryan, > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same > > > > > > observation, > > > > > > > >>>>>>>> had a > > > > > > > >>>>>>>>>>>>>>>> short > > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an > > > > > introduction > > > > > > > >>>>>> if > > > > > > > >>>>>>>>>>>>>>> someone > > > > > > > >>>>>>>>>>>>> joins > > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes > sense. > > > > > > > >>>>>> Especially > > > > > > > >>>>>>>>> when > > > > > > > >>>>>>>>>>>>>>> we > > > > > > > >>>>>>>>>>>>> also > > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a > reasonable > > > step. > > > > > > > >>>>>>>> Concepts > > > > > > > >>>>>>>>>>>>>>> still > > > > > > > >>>>>>>>>>>>> can > > > > > > > >>>>>>>>>>>>>>> stay the same. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding below > the > > > desk > > > > > > with > > > > > > > >>>>>>>> you > > > > > > > >>>>>>>>> and > > > > > > > >>>>>>>>>>>>>>>> +1 > > > > > > > >>>>>>>>>>>>> for > > > > > > > >>>>>>>>>>>>>>> raising the discussion. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep > > details > > > of > > > > > > > >>>>>> python > > > > > > > >>>>>>>>> names > > > > > > > >>>>>>>>>>>>>>>> the > > > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAG… but > > user > > > > > > facing > > > > > > > >>>>>> it > > > > > > > >>>>>>>>> is a > > > > > > > >>>>>>>>>>>>> workflow. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> Jens > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter < > > > > > > > >>>>>>>>>> ryan.hat...@astronomer.io <mailto: > > > ryan.hat...@astronomer.io > > > > > > > > > > > > > >>>>>>>>>>>>> .invalid> > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at > least > > > for > > > > > > now. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to > > Airflow > > > > > > users. > > > > > > > >>>>>>>>> Indeed, > > > > > > > >>>>>>>>>>>>>>>>> it > > > > > > > >>>>>>>>>>>>> has > > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians > > and > > > > > > > >>>>>> software > > > > > > > >>>>>>>>>>>>>>>>> engineers > > > > > > > >>>>>>>>>>>>> for > > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter. > > For > > > > > > someone > > > > > > > >>>>>>>> new > > > > > > > >>>>>>>>> to > > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, > one > > > of > > > > > the > > > > > > > >>>>>>>> first > > > > > > > >>>>>>>>>>>>>>>>> questions they > > > > > > > >>>>>>>>>>>>>>> will > > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The > > > answer > > > > > is > > > > > > > >>>>>>>> almost > > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You > > don't > > > > > need > > > > > > to > > > > > > > >>>>>>>>> worry > > > > > > > >>>>>>>>>>>>>>>>> about what > > > > > > > >>>>>>>>>>>>> that > > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow." > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a > > > couple > > > > > > > >>>>>>>> important > > > > > > > >>>>>>>>>>>> reasons: > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned > above, > > > "DAG" > > > > > > is > > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We > > want > > > > > > Airflow > > > > > > > >>>>>>>> to > > > > > > > >>>>>>>>> be > > > > > > > >>>>>>>>>>>>>>>>> approachable, and > > > > > > > >>>>>>>>>>>>>>> using > > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat > > > creates > > > > > > an > > > > > > > >>>>>>>>> initial > > > > > > > >>>>>>>>>>>>>>> barrier to > > > > > > > >>>>>>>>>>>>>>>>> understanding. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow > Concepts*: > > > The > > > > > DAG > > > > > > > >>>>>> is > > > > > > > >>>>>>>>> just > > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The > > > workflow > > > > > > > >>>>>> includes > > > > > > > >>>>>>>>> its > > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other > > > > > parameters, > > > > > > > >>>>>> and > > > > > > > >>>>>>>>> other > > > > > > > >>>>>>>>>>>>>>>>> metadata that > > > > > > > >>>>>>>>>>>>> the > > > > > > > >>>>>>>>>>>>>>>>> DAG component doesn’t account for. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow > > homepage > > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > > > > > https://airflow.apache.org/>>. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Apache Airflow® is a platform created by the > > > > > community > > > > > > to > > > > > > > >>>>>>>>>>>>>>> programmatically > > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows. > > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" > docs > > > page > > > > > > > >>>>>>>>>>>>>>>>> < > > > > > > > >>>>>>>>>> > > > > > > > >> > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html < > > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html> > > > > > > > >>>>>>>>>>>>>>>>>> , > > > > > > > >>>>>>>>>>>>> we > > > > > > > >>>>>>>>>>>>>>> can > > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is > > without > > > > > using > > > > > > > >>>>>>>> "DAG." > > > > > > > >>>>>>>>>>>>>>>>> It's > > > > > > > >>>>>>>>>>>>>>> only in > > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is > > > > > introduced > > > > > > > >>>>>> out > > > > > > > >>>>>>>> of > > > > > > > >>>>>>>>>>>>>>>>> nowhere > > > > > > > >>>>>>>>>>>>>>> as a > > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it: > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection > of > > > tasks > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these > > > > > > > >>>>>> introductions > > > > > > > >>>>>>>> to > > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate > > > DAGs; > > > > > it > > > > > > > >>>>>>>>>>> orchestrates > > > > > > > >>>>>>>>>>>> workflows*. > > > > > > > >>>>>>>>>>>>>>> The > > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant > > to > > > > > almost > > > > > > > >>>>>>>> every > > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to. > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for > the > > > term > > > > > > "DAG" > > > > > > > >>>>>>>> and > > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with > > > > > > "workflow". > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> For example, instead of... > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> @dag( > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > > > > >>>>>>>>>>>>>>>>> ... > > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > > > > > > >>>>>>>>>>>>>>>>> ) > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> Users could do... > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> @workflow( > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > > > > >>>>>>>>>>>>>>>>> ... > > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > > > > > > >>>>>>>>>>>>>>>>> ) > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away. > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >> > > > ------------------------------------------------------------------ > > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > > > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > > > > dev-unsubscr...@airflow.apache.org> > > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail: > > > > > > > >>>>>>>> dev-h...@airflow.apache.org <mailto: > > > > > dev-h...@airflow.apache.org> > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>> > > > ------------------------------------------------------------------- > > > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail: > > > > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > > > > dev-unsubscr...@airflow.apache.org> > > > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail: > > > > > > > >>>>>> dev-h...@airflow.apache.org <mailto: > > > dev-h...@airflow.apache.org > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>> > > > > > > --------------------------------------------------------------------- > > > > > > > >>>>>>>>>>>> To unsubscribe, e-mail: > > > > > dev-unsubscr...@airflow.apache.org <mailto: > > > > > dev-unsubscr...@airflow.apache.org> > > > > > > > >>>>>>>>>>>> For additional commands, e-mail: > > > > > > dev-h...@airflow.apache.org <mailto:dev-h...@airflow.apache.org> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>>> > > > > > > > >>>>>>>>>>> > > > > > > > >>>>>>>>>> > > > > > > > >>>>>>>>> > > > > > > > >>>>>>>> > > > > > > > >>>>>>> > > > > > > > >>>>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > <mailto: > > > > > dev-unsubscr...@airflow.apache.org> > > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > <mailto: > > > > > dev-h...@airflow.apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
