I think my biggest concern is a marketing one and not a technical one. As has been mentioned on the thread the terms airflow and dag are kind of synonymous and I certainly don’t want to give the impression that we are breaking more than we are breaking.
I wouldn’t die on this hill, but I’m slightly concerned. On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> wrote: > I’m not sure about adding ruff rules here 🤔 I think ruff rules are best > suited for user-facing things but not the airflow code base itself. If what > we mean is adding a rule to avoid users using "DAG" *after* we rename it, > it's definitely a +1000. > > I just created GitHub issues for this removing "DAG" > > https://github.com/apache/airflow/issues/46842 > https://github.com/apache/airflow/issues/46843 > > One thing I'm not sure about is whether we want to get rid of `from > airflow import DAG` as well. 🤔 > > Best, > Wei > > On 2025/02/17 19:50:57 Jarek Potiuk wrote: > > Hard to say until it's looked at :) > > > > On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu <aritrabasu1...@gmail.com> > > wrote: > > > > > I can take it up, it's mostly just a doc update right? Or are we doing > code > > > files replacement too? > > > -- > > > Regards, > > > Aritra Basu > > > > > > On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, <ja...@potiuk.com> wrote: > > > > > > > Sounds like another +1000 files big PR is coming :) ? Any volunteers > to > > > > make it ? It's fun. > > > > > > > > On Mon, Feb 17, 2025 at 6:52 PM Omkar P <droiddev5...@gmail.com> > wrote: > > > > > > > > > +1 for ruff rules :) > > > > > > > > > > Also would be nice to introduce 'Dag' to replace 'DAG' in the > Airflow > > > > > docs, in line with the new UI changes and to make the renaming > > > > > consistent across user-facing pages. > > > > > > > > > > Regards, > > > > > Omkar > > > > > > > > > > On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > > > > > +1 :) . Maybe we could add a ruff rule for that :) > > > > > > > > > > > > On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> > wrote: > > > > > > > > > > > > > It seems that our current conclusion is to use "dag" or "Dag" > > > instead > > > > > of > > > > > > > "DAG" whenever possible. Should we replace all "DAG" in the > > > codebase > > > > > with > > > > > > > "dag" or "Dag"? If it's too late for that (which it might be > 🤔), > > > > > should > > > > > > we > > > > > > > at least avoid introducing new "DAG" in the following PRs? > > > > > > > > > > > > > > Best, > > > > > > > Wei > > > > > > > > > > > > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > > > > > > > Here's a PR to use "dag" as a word in the new 3.0 UI: > > > > > > > > https://github.com/apache/airflow/pull/43325 > > > > > > > > Let me know if that's the direction we want to go. > > > > > > > > > > > > > > > > On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > > > > > > > > <rbish...@amazon.com.invalid> wrote: > > > > > > > > > > > > > > > > > I think Brent & Daniel summarized it best, "dag" is > synonymous > > > > with > > > > > > > > > workflows in Airflow through the way we talk and explain > what > > > > > Airflow > > > > > > > is > > > > > > > > > all about. Although folks would ask, I don’t ever use the > > > > > > mathematical > > > > > > > > > definition of DAG. > > > > > > > > > It will be challenging and possibly confusing for many > users > > > > making > > > > > > > such a > > > > > > > > > change - I would rather direct that energy to appending the > > > > Oxford > > > > > > > > > definition of "dag" to include a reference to workflows in > > > > Airflow. > > > > > > __ > > > > > > > > > > > > > > > > > > -- Rajesh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com > > > > <mailto: > > > > > > > > > ja...@potiuk.com>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > CAUTION: This email originated from outside of the > > > organization. > > > > Do > > > > > > not > > > > > > > > > click links or open attachments unless you can confirm the > > > sender > > > > > and > > > > > > > know > > > > > > > > > the content is safe. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un > > > expéditeur > > > > > > > externe. > > > > > > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > si > > > vous > > > > > ne > > > > > > > pouvez > > > > > > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > > > > certain > > > > > > > que > > > > > > > > > le contenu ne présente aucun risque. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From Guido's post: > > > > > > > > > > > > > > > > > > > > > > > > > > > "Around this time the renaming seems to have been renamed". > > > > > > > > > > > > > > > > > > > > > > > > > > > Naming is hard. > > > > > > > > > > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P < > > > droiddev5...@gmail.com > > > > > > > <mailto: > > > > > > > > > droiddev5...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, completely agree with above comments, dag is an > Airflow > > > > term > > > > > > now > > > > > > > > > > rather > > > > > > > > > > than just a "directed acyclic graph". Believe it or not, > I've > > > > > > worked > > > > > > > with > > > > > > > > > > work > > > > > > > > > > folks who've been using dags in Airflow for years now > and are > > > > pro > > > > > > > devs > > > > > > > > > but > > > > > > > > > > have trouble remembering the full form of a DAG! For > them, > > > dag > > > > = > > > > > > > Airflow. > > > > > > > > > > > > > > > > > > > > workflow, pipeline, flow are surely better terms but > will be > > > a > > > > > > major > > > > > > > > > > behavior > > > > > > > > > > change for the developer community, so we'll need a solid > > > plan > > > > on > > > > > > > how to > > > > > > > > > > introduce it, when we do. > > > > > > > > > > > > > > > > > > > > While we discuss this, I'd like to share about the Great > > > > Renaming > > > > > > in > > > > > > > core > > > > > > > > > > Python started by Guido back in 2009. We could probably > get > > > > some > > > > > > > > > learnings > > > > > > > > > > from there? Who knows! > > > > > > > > > > > > > > > > > > > > Guido's blog (2009): > > > > > > > > > > > > > > > > > > > > > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > > > > > < > > > > > > > > > > > > > > > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > > > > > > > > > > > > > > > > Follow-up discussion (2024): > > > > > > > > > > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > > < > > > > > > > > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Omkar > > > > > > > > > > > > > > > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < > weilee...@gmail.com > > > > > > > <mailto: > > > > > > > > > weilee...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > > > > > I think we should probably just accept it as an airflow > > > term. > > > > > At > > > > > > > least, > > > > > > > > > > > that’s how I understood it when I first used Airflow. I > > > feel > > > > > > > renaming > > > > > > > > > it > > > > > > > > > > at > > > > > > > > > > > this stage would require considerable effort from > > > maintainers > > > > > and > > > > > > > > > > existing > > > > > > > > > > > users without providing equivalent benefits. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Wei > > > > > > > > > > > > > > > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik < > > > > kaxiln...@gmail.com > > > > > > > > > <mailto:kaxiln...@gmail.com>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle > > > this > > > > > > > discussion > > > > > > > > > > for > > > > > > > > > > > > Airflow 4 :) -- but right now it will cause too much > > > > > disruption > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > > > > > > > > > > > <consta...@astronomer.io.inva <mailto: > > > > > > > consta...@astronomer.io.inva>lid> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> In my experience, when you ask those with Airflow > > > > experience > > > > > > > what a > > > > > > > > > > dag > > > > > > > > > > > is, > > > > > > > > > > > >> they'll start talking about workflow attributes - > stuff > > > > like > > > > > > > dags > > > > > > > > > > being > > > > > > > > > > > a > > > > > > > > > > > >> series of steps or tasks with owners. The structure > > > > doesn't > > > > > > > come up. > > > > > > > > > > > >> > > > > > > > > > > > >> Echo-ing others, at this point, my vote is to > embrace > > > the > > > > > name > > > > > > > and > > > > > > > > > > > >> de-emphasize the mathematical structure aspect. > > > > > > > > > > > >> > > > > > > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > > > > > > > > > > <vik...@astronomer.io.inva <mailto: > > > vik...@astronomer.io.inva > > > > > > >lid> > > > > > > > > > > > >> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >>> It's an interesting discussion and I remember > > > struggling > > > > > with > > > > > > > this > > > > > > > > > > > when I > > > > > > > > > > > >>> started working with Airflow. > > > > > > > > > > > >>> But, I also agree with the viewpoint of it being an > > > > > > established > > > > > > > > > > concept > > > > > > > > > > > >> now > > > > > > > > > > > >>> regardless of the origin. > > > > > > > > > > > >>> > > > > > > > > > > > >>> I am personally leaning towards the perspective > best > > > > > > expressed > > > > > > > by > > > > > > > > > > > Daniel > > > > > > > > > > > >>> Standish and Brent of using Dag as a word, rather > than > > > > the > > > > > > > computer > > > > > > > > > > > >> science > > > > > > > > > > > >>> concept. > > > > > > > > > > > >>> > > > > > > > > > > > >>> Best regards, > > > > > > > > > > > >>> Vikram > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > > > > > > > > > > >> <oniko...@amazon.com.inva <mailto: > > > > oniko...@amazon.com.inva > > > > > > >lid > > > > > > > > > > > >>>> > > > > > > > > > > > >>> wrote: > > > > > > > > > > > >>> > > > > > > > > > > > >>>> I agree with the general sentiment of: You're > right > > > > Ryan, > > > > > > DAG > > > > > > > > > isn't > > > > > > > > > > > >> great > > > > > > > > > > > >>>> and I'd rather workflow, but changing it will > cause > > > much > > > > > > more > > > > > > > > > > wreckage > > > > > > > > > > > >>> than > > > > > > > > > > > >>>> it solves. > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> Also agree with the idea to just move away from > > > defining > > > > > > DAG. > > > > > > > I > > > > > > > > > > think > > > > > > > > > > > >>>> we've been naturally doing that as a community > for a > > > > while > > > > > > now > > > > > > > > > > anyway, > > > > > > > > > > > >> so > > > > > > > > > > > >>>> that feels like a natural step. > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> Cheers, > > > > > > > > > > > >>>> Niko > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> ________________________________ > > > > > > > > > > > >>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: > > > > > > > a...@apache.org>> > > > > > > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > > > > > > > > > > >>>> To: dev@airflow.apache.org <mailto: > > > > dev@airflow.apache.org > > > > > > > > > > > > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the > term > > > > "DAG" > > > > > > > for end > > > > > > > > > > > >> users > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> CAUTION: This email originated from outside of the > > > > > > > organization. > > > > > > > > > Do > > > > > > > > > > > not > > > > > > > > > > > >>>> click links or open attachments unless you can > confirm > > > > the > > > > > > > sender > > > > > > > > > > and > > > > > > > > > > > >>> know > > > > > > > > > > > >>>> the content is safe. > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> AVERTISSEMENT: Ce courrier électronique provient > d’un > > > > > > > expéditeur > > > > > > > > > > > >> externe. > > > > > > > > > > > >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > > > jointe > > > > > si > > > > > > > vous > > > > > > > > > ne > > > > > > > > > > > >>> pouvez > > > > > > > > > > > >>>> pas confirmer l’identité de l’expéditeur et si > vous > > > > n’êtes > > > > > > pas > > > > > > > > > > certain > > > > > > > > > > > >>> que > > > > > > > > > > > >>>> le contenu ne présente aucun risque. > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> Best argument in favour of keeping “dags” as a > term — > > > > > > getting > > > > > > > to > > > > > > > > > > > re-use > > > > > > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > > > > > > > > https://i.imgflip.com/1xhtwh.jpg> > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> In all seriousness: I don’t mind either way, both > > > sides > > > > > have > > > > > > > good > > > > > > > > > > > >> reasons > > > > > > > > > > > >>>> presented. > > > > > > > > > > > >>>> > > > > > > > > > > > >>>> -a > > > > > > > > > > > >>>> > > > > > > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > > > > > > > > > > >>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > > > > > > daniel.stand...@astronomer.io.inva>LID> wrote: > > > > > > > > > > > >>>>> > > > > > > > > > > > >>>>> Yeah just say, when asked where the name comes > from, > > > > > "well, > > > > > > > no > > > > > > > > > one > > > > > > > > > > > >>>> actually > > > > > > > > > > > >>>>> knows but..." and then make something up. > > > > > > > > > > > >>>>> > > > > > > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < > > > > > > > ja...@potiuk.com > > > > > > > > > <mailto:ja...@potiuk.com>> > > > > > > > > > > > >> wrote: > > > > > > > > > > > >>>>> > > > > > > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is > the > > > > > > > > > tongue-twister, > > > > > > > > > > > >>>>>> > > > > > > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < > > > > > > > ja...@potiuk.com > > > > > > > > > <mailto:ja...@potiuk.com>> > > > > > > > > > > > >>> wrote: > > > > > > > > > > > >>>>>> > > > > > > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I > would > > > very > > > > > > much > > > > > > > want > > > > > > > > > > to > > > > > > > > > > > >> be > > > > > > > > > > > >>>>>> able > > > > > > > > > > > >>>>>>> to say just "dag" without explaining it > further. > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> For me every time I explain "DAG" at a talk > it's a > > > > > > > > > > tongue-twister, > > > > > > > > > > > >>> and > > > > > > > > > > > >>>> I > > > > > > > > > > > >>>>>>> almost stutter on trying to recall how to > pronounce > > > > it > > > > > > > > > properly. > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> J. > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > > > > > > > > > > >>>>>> <br...@astronomer.io.inva <mailto: > > > > > > br...@astronomer.io.inva > > > > > > > >lid> > > > > > > > > > > > >>>>>>> wrote: > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>>> I remember we explored renaming "DAG" when > > > starting > > > > on > > > > > > > AIP-38 > > > > > > > > > to > > > > > > > > > > > >>>>>> modernize > > > > > > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more > > > > > > > descriptive of > > > > > > > > > > what > > > > > > > > > > > >>> one > > > > > > > > > > > >>>>>> is > > > > > > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph > is an > > > > > > > > > implementation > > > > > > > > > > > >>>> detail. > > > > > > > > > > > >>>>>>>> But I agree with Daniel Standish, at this > point > > > > "DAG" > > > > > > has > > > > > > > > > become > > > > > > > > > > > >>> "dag" > > > > > > > > > > > >>>>>> , a > > > > > > > > > > > >>>>>>>> word in its own right. > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > >>>>>>>> Examples for "dag" are abound in community > > > > discussion, > > > > > > > Airflow > > > > > > > > > > > >>> Summit > > > > > > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's > > > > embrace > > > > > > > "dag". > > > > > > > > > A > > > > > > > > > > > >> user > > > > > > > > > > > >>>>>> just > > > > > > > > > > > >>>>>>>> needs to learn one new word vs the technical > > > concept > > > > > > > behind > > > > > > > > > that > > > > > > > > > > > >>>> word. I > > > > > > > > > > > >>>>>>>> think that is much less effort than > refactoring so > > > > > much > > > > > > > code, > > > > > > > > > > > >>>>>>>> documentation, blog posts, stack overflow > > > questions, > > > > > > etc. > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel > Standish > > > > > > > > > > > >>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > > > > > > > > daniel.stand...@astronomer.io.inva>lid> wrote: > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot > of > > > > pain > > > > > > for > > > > > > > > > > > >>>> questionable > > > > > > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the > idea. I > > > > > agree > > > > > > > the > > > > > > > > > > > >>>>>>>> association > > > > > > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful. > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> And along those lines, I offer here some less > > > > > invasive > > > > > > > > > > > >> mitigations. > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> One thing we can do no matter what is to > > > > de-emphasize > > > > > > the > > > > > > > > > math > > > > > > > > > > > >> nerd > > > > > > > > > > > >>>>>>>> origins > > > > > > > > > > > >>>>>>>>> of the name. That is to say, in docs / > website / > > > > etc, > > > > > > > *never > > > > > > > > > > > >>> define* > > > > > > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic > > > > graph. > > > > > > > Always > > > > > > > > > > > >> define > > > > > > > > > > > >>>> it > > > > > > > > > > > >>>>>>>> as a > > > > > > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc. > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is > like a > > > > > > > historical > > > > > > > > > > > >>>> footnote, > > > > > > > > > > > >>>>>>>> and > > > > > > > > > > > >>>>>>>>> we could make one mention of it somewhere > hidden. > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> We could also start using lowercase in the > docs > > > in > > > > > > > general > > > > > > > > > e.g. > > > > > > > > > > > >>>>>> writing > > > > > > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / > "DAGs" > > > > etc. > > > > > > The > > > > > > > > > > upper > > > > > > > > > > > >>> case > > > > > > > > > > > >>>>>>>> part > > > > > > > > > > > >>>>>>>>> of it makes it look like an acronym; but > "dag" in > > > > > > > airlfow is > > > > > > > > > > just > > > > > > > > > > > >>> an > > > > > > > > > > > >>>>>>>>> airflow concept and the association with > "DAGs" > > > is > > > > > not > > > > > > > really > > > > > > > > > > > >>>>>> unhelpful. > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow > is > > > its > > > > > own > > > > > > > > > thing, > > > > > > > > > > is > > > > > > > > > > > >>>>>>>>> *not* strictly > > > > > > > > > > > >>>>>>>>> speaking a directed acyclic graph (which > nobody > > > > knows > > > > > > > about > > > > > > > > > > > >>> anyway), > > > > > > > > > > > >>>>>> and > > > > > > > > > > > >>>>>>>>> tell them what it is in simple terms that > normal > > > > > people > > > > > > > > > > > >> understand. > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk > < > > > > > > > > > ja...@potiuk.com <mailto:ja...@potiuk.com> > > > > > > > > > > > > > > > > > > > > > > >>>>>> wrote: > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it > will > > > be > > > > > > > extremely > > > > > > > > > > > >>>>>>>> difficult to > > > > > > > > > > > >>>>>>>>>> get rid of it completely. Also I think it > will > > > > make > > > > > a > > > > > > > lot of > > > > > > > > > > > >>>>>> "google" > > > > > > > > > > > >>>>>>>>>> searches and "stack overflow" searches not > > > finding > > > > > the > > > > > > > right > > > > > > > > > > > >>>>>> answers. > > > > > > > > > > > >>>>>>>>> This > > > > > > > > > > > >>>>>>>>>> is one of the strengths of Airflow - > besides the > > > > > > > community > > > > > > > > > and > > > > > > > > > > > >>> ideas > > > > > > > > > > > >>>>>>>> that > > > > > > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of > > > examples, > > > > > > > problems > > > > > > > > > and > > > > > > > > > > > >>>>>>>> solutions > > > > > > > > > > > >>>>>>>>>> you can so easily find (and we have to > remember > > > > that > > > > > > > all the > > > > > > > > > > AI > > > > > > > > > > > >>>>>>>> trained > > > > > > > > > > > >>>>>>>>> on > > > > > > > > > > > >>>>>>>>>> past data will be also rather poorly > matching > > > > > queries > > > > > > of > > > > > > > > > > people. > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily > > > > switch. > > > > > > > And if > > > > > > > > > we > > > > > > > > > > > >> do > > > > > > > > > > > >>> - > > > > > > > > > > > >>>>>> I > > > > > > > > > > > >>>>>>>>>> would be for using workflow or pipeline > instead > > > of > > > > > > > `dag` if > > > > > > > > > > not > > > > > > > > > > > >>> the > > > > > > > > > > > >>>>>>>> above > > > > > > > > > > > >>>>>>>>>> reason, but I think I am here with Igor > that it > > > > > might > > > > > > > cause > > > > > > > > > > more > > > > > > > > > > > >>>>>>>> problems > > > > > > > > > > > >>>>>>>>>> than it solves. > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >>>>>>>>>> But I am not 100% against - if others will > think > > > > > it's > > > > > > a > > > > > > > good > > > > > > > > > > > >>> idea, I > > > > > > > > > > > >>>>>>>> am > > > > > > > > > > > >>>>>>>>> ok > > > > > > > > > > > >>>>>>>>>> with it. > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >>>>>>>>>> J, > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek > Bhakat > > > > > > > > > > > >>>>>>>>>> <abhishek.bha...@astronomer.io.inva > <mailto: > > > > > > > > > abhishek.bha...@astronomer.io.inva>lid> wrote: > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less > sense > > > to > > > > > > > someone > > > > > > > > > new > > > > > > > > > > > >> to > > > > > > > > > > > >>>>>>>>> workflow > > > > > > > > > > > >>>>>>>>>>> orchestration. But it does also show the > nature > > > > of > > > > > > > being > > > > > > > > > > > >> acyclic. > > > > > > > > > > > >>>>>>>> Sure, > > > > > > > > > > > >>>>>>>>>> as > > > > > > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to > workaround it. > > > > > > Still, > > > > > > > in > > > > > > > > > my > > > > > > > > > > > >>>>>>>> opinion, > > > > > > > > > > > >>>>>>>>>> there > > > > > > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in > > > > > workflow > > > > > > > > > > > >>>>>> orchestration. > > > > > > > > > > > >>>>>>>>> Most > > > > > > > > > > > >>>>>>>>>>> (*if > > > > > > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be > > > covered > > > > > > > using an > > > > > > > > > > > >>> acyclic > > > > > > > > > > > >>>>>>>>> manner > > > > > > > > > > > >>>>>>>>>>> with multiple runs. Hence, the > idempotency. So > > > I > > > > > > would > > > > > > > want > > > > > > > > > > the > > > > > > > > > > > >>>>>>>>> "acyclic" > > > > > > > > > > > >>>>>>>>>>> word to stick. > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>> Regards, > > > > > > > > > > > >>>>>>>>>>> Avi > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > > > > > > > > bernd.stroe...@kosakya.de <mailto: > bernd.stroe...@kosakya.de>> > > > > > > > > > > > >>>>>> wrote: > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an > > > Airflow > > > > > Fan; > > > > > > > so > > > > > > > > > many > > > > > > > > > > > >> new > > > > > > > > > > > >>>>>>>>> ideas. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be > > > > replaced > > > > > by > > > > > > > the > > > > > > > > > > more > > > > > > > > > > > >>>>>>>> general > > > > > > > > > > > >>>>>>>>>>> Term > > > > > > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow > > > > (Petri) > > > > > > Net > > > > > > > > > (AFN) > > > > > > > > > > > >>>>>> (maybe > > > > > > > > > > > >>>>>>>>>>> without > > > > > > > > > > > >>>>>>>>>>>> a direction); > > > > > > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a > > > Graph > > > > > > > Database. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow > Graph > > > > (AFG) > > > > > > > might be > > > > > > > > > > > >>>>>>>> assigned > > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > > >>>>>>>>>> an > > > > > > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member > of a > > > > > > library. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> A running Graph might have a different > > > structure > > > > > > than > > > > > > > a > > > > > > > > > > > >>>>>>>> configuration > > > > > > > > > > > >>>>>>>>>>>> Graph. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Best Regards > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Bernd Ströhle > > > > > > > > > > > >>>>>>>>>>>> M: +49 171 5357916 > > > > > > > > > > > >>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > > > > > > > > bernd.stroe...@gmail.com> > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> -----Original Message----- > > > > > > > > > > > >>>>>>>>>>>> From: Igor Kholopov > <ikholo...@google.com.inva > > > > > > > <mailto: > > > > > > > > > ikholo...@google.com.inva>LID> > > > > > > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > > > > > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > > > > > > dev@airflow.apache.org > > > > > > > > > > > > > > > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the > term > > > > > "DAG" > > > > > > > for > > > > > > > > > end > > > > > > > > > > > >>>>>> users > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly > > > > suboptimal, > > > > > it > > > > > > > is > > > > > > > > > part > > > > > > > > > > > >> of > > > > > > > > > > > >>>>>>>>> Airflow > > > > > > > > > > > >>>>>>>>>>>> DAG definition interface at so many > levels, > > > that > > > > > any > > > > > > > > > attempt > > > > > > > > > > > >> to > > > > > > > > > > > >>>>>>>>> change > > > > > > > > > > > >>>>>>>>>> it > > > > > > > > > > > >>>>>>>>>>>> will only introduce more chaos, not > reduce it. > > > > The > > > > > > > only > > > > > > > > > > thing > > > > > > > > > > > >>>>>>>> that is > > > > > > > > > > > >>>>>>>>>>> worse > > > > > > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is > when > > > > > there > > > > > > > are > > > > > > > > > two > > > > > > > > > > > >> ways > > > > > > > > > > > >>>>>>>> to > > > > > > > > > > > >>>>>>>>>>> define > > > > > > > > > > > >>>>>>>>>>>> the same thing. Countless articles and > > > tutorials > > > > > > will > > > > > > > > > > suddenly > > > > > > > > > > > >>>>>>>> become > > > > > > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows > as > > > > > "DAG"s. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> We are already at risk of scaring the > users > > > away > > > > > > with > > > > > > > a > > > > > > > > > > number > > > > > > > > > > > >>>>>> of > > > > > > > > > > > >>>>>>>>>>> breaking > > > > > > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more > > > > breaking > > > > > > > changes > > > > > > > > > > for > > > > > > > > > > > >>>>>> the > > > > > > > > > > > >>>>>>>>> most > > > > > > > > > > > >>>>>>>>>>>> basic things is not something that people > are > > > > > > looking > > > > > > > for. > > > > > > > > > > > >>>>>>>> Attempting > > > > > > > > > > > >>>>>>>>>> to > > > > > > > > > > > >>>>>>>>>>>> change the fundamental terms will be > > > interpreted > > > > > as > > > > > > an > > > > > > > > > even > > > > > > > > > > > >>>>>>>> stronger > > > > > > > > > > > >>>>>>>>>>> signal > > > > > > > > > > > >>>>>>>>>>>> of project immaturity. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing > the > > > > term > > > > > > in > > > > > > > the > > > > > > > > > > long > > > > > > > > > > > >>>>>>>> run. I > > > > > > > > > > > >>>>>>>>>>> even > > > > > > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating > it in > > > > the > > > > > > DAG > > > > > > > > > > > >> definition > > > > > > > > > > > >>>>>>>>>>> interface. > > > > > > > > > > > >>>>>>>>>>>> We better put our time and efforts in > other > > > > places > > > > > > in > > > > > > > > > > Airflow, > > > > > > > > > > > >>>>>> of > > > > > > > > > > > >>>>>>>>> which > > > > > > > > > > > >>>>>>>>>>>> there are plenty. > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> Kind regards, > > > > > > > > > > > >>>>>>>>>>>> Igor > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas > Harenslak > > > > > > > > > > > >>>>>>>>>> <b...@astronomer.io.inva <mailto: > > > > > > b...@astronomer.io.inva > > > > > > > >lid > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>> Couple of thoughts: > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of “DAG” > have > > > > > already > > > > > > > faded > > > > > > > > > > over > > > > > > > > > > > >>>>>>>> time, > > > > > > > > > > > >>>>>>>>>>>>> for example there are now several ways to > > > > create > > > > > > > cyclic > > > > > > > > > > > >>>>>> graphs, > > > > > > > > > > > >>>>>>>>> e.g. > > > > > > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine > > > these > > > > > > > > > properties > > > > > > > > > > > >>>>>>>>> vanishing > > > > > > > > > > > >>>>>>>>>>>>> even more in the future, so from that > > > > > perspective I > > > > > > > > > support > > > > > > > > > > > >>>>>>>>> changing > > > > > > > > > > > >>>>>>>>>>>>> “DAG" to a more generic name. > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do > > > > naming: > > > > > > > > > > > >>>>>>>>>>>>> Dagster: pipeline > > > > > > > > > > > >>>>>>>>>>>>> Prefect: flow > > > > > > > > > > > >>>>>>>>>>>>> Flyte: workflow > > > > > > > > > > > >>>>>>>>>>>>> Temporal: workflow > > > > > > > > > > > >>>>>>>>>>>>> Kestra: flow > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>> I think “workflow” is the most fitting > name. > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this > change, I > > > > > suggest > > > > > > > > > > defining > > > > > > > > > > > >> a > > > > > > > > > > > >>>>>>>>> clear > > > > > > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce > the > > > > > > > deprecation in > > > > > > > > > > > >>>>>>>> Airflow > > > > > > > > > > > >>>>>>>>> 3, > > > > > > > > > > > >>>>>>>>>>>>> and remove “DAG” in Airflow 4? > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>> Bas > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < > > > > > > neil4r...@gmail.com > > > > > > > > > <mailto:neil4r...@gmail.com>> > > > > > > > > > > wrote: > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, > > > > > > especially > > > > > > > when > > > > > > > > > > > >>>>>> most > > > > > > > > > > > >>>>>>>>> other > > > > > > > > > > > >>>>>>>>>>>>>> platforms embrace the term > wholeheartedly. > > > > > > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or > > > confusing > > > > > > > about it > > > > > > > > > at > > > > > > > > > > > >>>>>>>> all, > > > > > > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly > > > > > confusing > > > > > > > to > > > > > > > > > most > > > > > > > > > > > >>>>>>>> users > > > > > > > > > > > >>>>>>>>>> who > > > > > > > > > > > >>>>>>>>>>>>>> have been > > > > > > > > > > > >>>>>>>>>>>>> using > > > > > > > > > > > >>>>>>>>>>>>>> the term for years. > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping > > > Chung > > > > > > > > > > > >>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: > > > > > > > t...@astronomer.io.inva > > > > > > > > > >lid > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with > the > > > term > > > > > > DAG. > > > > > > > The > > > > > > > > > > only > > > > > > > > > > > >>>>>>>>>> problem > > > > > > > > > > > >>>>>>>>>>>>> (aside > > > > > > > > > > > >>>>>>>>>>>>>>> from actually telling people—including > > > > > myself—to > > > > > > > stop > > > > > > > > > > using > > > > > > > > > > > >>>>>>>> the > > > > > > > > > > > >>>>>>>>>>>>>>> term) > > > > > > > > > > > >>>>>>>>>>>>> is to > > > > > > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative. > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> I can’t recall who, but someone > mentioned > > > > > > > “workflow” is > > > > > > > > > > not > > > > > > > > > > > >>>>>>>> very > > > > > > > > > > > >>>>>>>>>>>>> accurate > > > > > > > > > > > >>>>>>>>>>>>>>> for Airflow. The term “definition” was > > > > > proposed, > > > > > > > but > > > > > > > > > > it’s a > > > > > > > > > > > >>>>>>>> bit > > > > > > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few > places > > > and > > > > > kept > > > > > > > > > finding > > > > > > > > > > > >>>>>>>> myself > > > > > > > > > > > >>>>>>>>>>>>>>> doubting “what definition?” and > wanting to > > > > > > clarify > > > > > > > “DAG > > > > > > > > > > > >>>>>>>>>> definition” > > > > > > > > > > > >>>>>>>>>>>>>>> (defeating the purpose). > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> TP > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens > Scheffler > > > > > > > > > > > >>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > > > > > > > > > j_scheff...@gmx.de.inva>LID> > > > > > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> Hi Ryan, > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the > exactly > > > same > > > > > > > > > > observation, > > > > > > > > > > > >>>>>>>> had a > > > > > > > > > > > >>>>>>>>>>>>>>>> short > > > > > > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is > always > > > an > > > > > > > > > introduction > > > > > > > > > > > >>>>>> if > > > > > > > > > > > >>>>>>>>>>>>>>> someone > > > > > > > > > > > >>>>>>>>>>>>> joins > > > > > > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming > makes > > > > > sense. > > > > > > > > > > > >>>>>> Especially > > > > > > > > > > > >>>>>>>>> when > > > > > > > > > > > >>>>>>>>>>>>>>> we > > > > > > > > > > > >>>>>>>>>>>>> also > > > > > > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a > > > > > reasonable > > > > > > > step. > > > > > > > > > > > >>>>>>>> Concepts > > > > > > > > > > > >>>>>>>>>>>>>>> still > > > > > > > > > > > >>>>>>>>>>>>> can > > > > > > > > > > > >>>>>>>>>>>>>>> stay the same. > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding > > > below > > > > > the > > > > > > > desk > > > > > > > > > > with > > > > > > > > > > > >>>>>>>> you > > > > > > > > > > > >>>>>>>>> and > > > > > > > > > > > >>>>>>>>>>>>>>>> +1 > > > > > > > > > > > >>>>>>>>>>>>> for > > > > > > > > > > > >>>>>>>>>>>>>>> raising the discussion. > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we > keep > > > > > > details > > > > > > > of > > > > > > > > > > > >>>>>> python > > > > > > > > > > > >>>>>>>>> names > > > > > > > > > > > >>>>>>>>>>>>>>>> the > > > > > > > > > > > >>>>>>>>>>>>>>> same because the execution is still a > DAG… > > > > but > > > > > > user > > > > > > > > > > facing > > > > > > > > > > > >>>>>> it > > > > > > > > > > > >>>>>>>>> is a > > > > > > > > > > > >>>>>>>>>>>>> workflow. > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> Jens > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan > Hatter < > > > > > > > > > > > >>>>>>>>>> ryan.hat...@astronomer.io <mailto: > > > > > > > ryan.hat...@astronomer.io > > > > > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> .invalid> > > > > > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your > swords... at > > > > > least > > > > > > > for > > > > > > > > > > now. > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little > meaning to > > > > > > Airflow > > > > > > > > > > users. > > > > > > > > > > > >>>>>>>>> Indeed, > > > > > > > > > > > >>>>>>>>>>>>>>>>> it > > > > > > > > > > > >>>>>>>>>>>>> has > > > > > > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some > > > > mathematicians > > > > > > and > > > > > > > > > > > >>>>>> software > > > > > > > > > > > >>>>>>>>>>>>>>>>> engineers > > > > > > > > > > > >>>>>>>>>>>>> for > > > > > > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually > > > > matter. > > > > > > For > > > > > > > > > > someone > > > > > > > > > > > >>>>>>>> new > > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow > > > orchestration, > > > > > one > > > > > > > of > > > > > > > > > the > > > > > > > > > > > >>>>>>>> first > > > > > > > > > > > >>>>>>>>>>>>>>>>> questions they > > > > > > > > > > > >>>>>>>>>>>>>>> will > > > > > > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a > DAG?" > > > > The > > > > > > > answer > > > > > > > > > is > > > > > > > > > > > >>>>>>>> almost > > > > > > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic > graph. > > > You > > > > > > don't > > > > > > > > > need > > > > > > > > > > to > > > > > > > > > > > >>>>>>>>> worry > > > > > > > > > > > >>>>>>>>>>>>>>>>> about what > > > > > > > > > > > >>>>>>>>>>>>> that > > > > > > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your > > > workflow." > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at > > > least > > > > a > > > > > > > couple > > > > > > > > > > > >>>>>>>> important > > > > > > > > > > > >>>>>>>>>>>> reasons: > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As > mentioned > > > > > above, > > > > > > > "DAG" > > > > > > > > > > is > > > > > > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and > confusing. > > > > We > > > > > > want > > > > > > > > > > Airflow > > > > > > > > > > > >>>>>>>> to > > > > > > > > > > > >>>>>>>>> be > > > > > > > > > > > >>>>>>>>>>>>>>>>> approachable, and > > > > > > > > > > > >>>>>>>>>>>>>>> using > > > > > > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right > off the > > > > bat > > > > > > > creates > > > > > > > > > > an > > > > > > > > > > > >>>>>>>>> initial > > > > > > > > > > > >>>>>>>>>>>>>>> barrier to > > > > > > > > > > > >>>>>>>>>>>>>>>>> understanding. > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow > > > > > Concepts*: > > > > > > > The > > > > > > > > > DAG > > > > > > > > > > > >>>>>> is > > > > > > > > > > > >>>>>>>>> just > > > > > > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow > workflow. The > > > > > > > workflow > > > > > > > > > > > >>>>>> includes > > > > > > > > > > > >>>>>>>>> its > > > > > > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen > > > other > > > > > > > > > parameters, > > > > > > > > > > > >>>>>> and > > > > > > > > > > > >>>>>>>>> other > > > > > > > > > > > >>>>>>>>>>>>>>>>> metadata that > > > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > > > >>>>>>>>>>>>>>>>> DAG component doesn’t account for. > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the > Airflow > > > > > > homepage > > > > > > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > > > > > > > > > https://airflow.apache.org/>>. > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Apache Airflow® is a platform > created by > > > > the > > > > > > > > > community > > > > > > > > > > to > > > > > > > > > > > >>>>>>>>>>>>>>> programmatically > > > > > > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor > workflows. > > > > > > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is > > > Airflow?" > > > > > docs > > > > > > > page > > > > > > > > > > > >>>>>>>>>>>>>>>>> < > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >> > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html < > > > > > > > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>>> , > > > > > > > > > > > >>>>>>>>>>>>> we > > > > > > > > > > > >>>>>>>>>>>>>>> can > > > > > > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what > Airflow is > > > > > > without > > > > > > > > > using > > > > > > > > > > > >>>>>>>> "DAG." > > > > > > > > > > > >>>>>>>>>>>>>>>>> It's > > > > > > > > > > > >>>>>>>>>>>>>>> only in > > > > > > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the > term > > > is > > > > > > > > > introduced > > > > > > > > > > > >>>>>> out > > > > > > > > > > > >>>>>>>> of > > > > > > > > > > > >>>>>>>>>>>>>>>>> nowhere > > > > > > > > > > > >>>>>>>>>>>>>>> as a > > > > > > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to > explain > > > it: > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a > > > collection > > > > > of > > > > > > > tasks > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs > in > > > > these > > > > > > > > > > > >>>>>> introductions > > > > > > > > > > > >>>>>>>> to > > > > > > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't > > > > orchestrate > > > > > > > DAGs; > > > > > > > > > it > > > > > > > > > > > >>>>>>>>>>> orchestrates > > > > > > > > > > > >>>>>>>>>>>> workflows*. > > > > > > > > > > > >>>>>>>>>>>>>>> The > > > > > > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons > > > > irrelevant > > > > > > to > > > > > > > > > almost > > > > > > > > > > > >>>>>>>> every > > > > > > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to. > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an > alias > > > for > > > > > the > > > > > > > term > > > > > > > > > > "DAG" > > > > > > > > > > > >>>>>>>> and > > > > > > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace > "DAG" > > > > with > > > > > > > > > > "workflow". > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> For example, instead of... > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> @dag( > > > > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > > > > > > > > >>>>>>>>>>>>>>>>> ... > > > > > > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > > > > > > > > > > >>>>>>>>>>>>>>>>> ) > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> Users could do... > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> @workflow( > > > > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily", > > > > > > > > > > > >>>>>>>>>>>>>>>>> ... > > > > > > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > > > > > > > > > > >>>>>>>>>>>>>>>>> ) > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running > > > away. > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > >> > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > > > > > > > > > > >>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > > > > > > > > dev-unsubscr...@airflow.apache.org> > > > > > > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail: > > > > > > > > > > > >>>>>>>> dev-h...@airflow.apache.org <mailto: > > > > > > > > > dev-h...@airflow.apache.org> > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > >>>>>>>>
