When I started contributing to Airflow, I had to read up on DAG, and now I know what it means. I wonder if we are about to have users who use the term Dag without knowing that there's a DAG.
Slightly concerned. -ephraim On Tue, 18 Feb 2025 at 10:26, Wei Lee <weilee...@gmail.com> wrote: > For user-facing things (e.g., `from airflow import DAG`), I’m more like > -0. But for documentation, docstring, internal things and etc., we probably > could still change most of them? > > Best, > Wei > > > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor <a...@apache.org> wrote: > > > > I also don’t personally think it’s worth the pain (not to mention > backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on that. > > > > -ash > > > >> On 18 Feb 2025, at 04:40, Daniel Imberman <daniel.imber...@gmail.com> > wrote: > >> > >> I think my biggest concern is a marketing one and not a technical one. > >> > >> As has been mentioned on the thread the terms airflow and dag are kind > of > >> synonymous and I certainly don’t want to give the impression that we are > >> breaking more than we are breaking. > >> > >> I wouldn’t die on this hill, but I’m slightly concerned. > >> > >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> wrote: > >> > >>> I’m not sure about adding ruff rules here 🤔 I think ruff rules are > best > >>> suited for user-facing things but not the airflow code base itself. If > what > >>> we mean is adding a rule to avoid users using "DAG" *after* we rename > it, > >>> it's definitely a +1000. > >>> > >>> I just created GitHub issues for this removing "DAG" > >>> > >>> https://github.com/apache/airflow/issues/46842 > >>> https://github.com/apache/airflow/issues/46843 > >>> > >>> One thing I'm not sure about is whether we want to get rid of `from > >>> airflow import DAG` as well. 🤔 > >>> > >>> Best, > >>> Wei > >>> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: > >>>> Hard to say until it's looked at :) > >>>> > >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu <aritrabasu1...@gmail.com > > > >>>> wrote: > >>>> > >>>>> I can take it up, it's mostly just a doc update right? Or are we > doing > >>> code > >>>>> files replacement too? > >>>>> -- > >>>>> Regards, > >>>>> Aritra Basu > >>>>> > >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, <ja...@potiuk.com> > wrote: > >>>>> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any volunteers > >>> to > >>>>>> make it ? It's fun. > >>>>>> > >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P <droiddev5...@gmail.com> > >>> wrote: > >>>>>> > >>>>>>> +1 for ruff rules :) > >>>>>>> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the > >>> Airflow > >>>>>>> docs, in line with the new UI changes and to make the renaming > >>>>>>> consistent across user-facing pages. > >>>>>>> > >>>>>>> Regards, > >>>>>>> Omkar > >>>>>>> > >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk <ja...@potiuk.com> > >>> wrote: > >>>>>>> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) > >>>>>>>> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org> > >>> wrote: > >>>>>>>> > >>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag" > >>>>> instead > >>>>>>> of > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the > >>>>> codebase > >>>>>>> with > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be > >>> 🤔), > >>>>>>> should > >>>>>>>> we > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs? > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Wei > >>>>>>>>> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: > >>>>>>>>>> https://github.com/apache/airflow/pull/43325 > >>>>>>>>>> Let me know if that's the direction we want to go. > >>>>>>>>>> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: > >>>>>>>>>> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is > >>> synonymous > >>>>>> with > >>>>>>>>>>> workflows in Airflow through the way we talk and explain > >>> what > >>>>>>> Airflow > >>>>>>>>> is > >>>>>>>>>>> all about. Although folks would ask, I don’t ever use the > >>>>>>>> mathematical > >>>>>>>>>>> definition of DAG. > >>>>>>>>>>> It will be challenging and possibly confusing for many > >>> users > >>>>>> making > >>>>>>>>> such a > >>>>>>>>>>> change - I would rather direct that energy to appending the > >>>>>> Oxford > >>>>>>>>>>> definition of "dag" to include a reference to workflows in > >>>>>> Airflow. > >>>>>>>> __ > >>>>>>>>>>> > >>>>>>>>>>> -- Rajesh > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" <ja...@potiuk.com > >>>>>> <mailto: > >>>>>>>>>>> ja...@potiuk.com>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> CAUTION: This email originated from outside of the > >>>>> organization. > >>>>>> Do > >>>>>>>> not > >>>>>>>>>>> click links or open attachments unless you can confirm the > >>>>> sender > >>>>>>> and > >>>>>>>>> know > >>>>>>>>>>> the content is safe. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un > >>>>> expéditeur > >>>>>>>>> externe. > >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > >>> si > >>>>> vous > >>>>>>> ne > >>>>>>>>> pouvez > >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes > >>> pas > >>>>>>>> certain > >>>>>>>>> que > >>>>>>>>>>> le contenu ne présente aucun risque. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> From Guido's post: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> "Around this time the renaming seems to have been renamed". > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Naming is hard. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> J. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < > >>>>> droiddev5...@gmail.com > >>>>>>>>> <mailto: > >>>>>>>>>>> droiddev5...@gmail.com>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an > >>> Airflow > >>>>>> term > >>>>>>>> now > >>>>>>>>>>>> rather > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not, > >>> I've > >>>>>>>> worked > >>>>>>>>> with > >>>>>>>>>>>> work > >>>>>>>>>>>> folks who've been using dags in Airflow for years now > >>> and are > >>>>>> pro > >>>>>>>>> devs > >>>>>>>>>>> but > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For > >>> them, > >>>>> dag > >>>>>> = > >>>>>>>>> Airflow. > >>>>>>>>>>>> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but > >>> will be > >>>>> a > >>>>>>>> major > >>>>>>>>>>>> behavior > >>>>>>>>>>>> change for the developer community, so we'll need a solid > >>>>> plan > >>>>>> on > >>>>>>>>> how to > >>>>>>>>>>>> introduce it, when we do. > >>>>>>>>>>>> > >>>>>>>>>>>> While we discuss this, I'd like to share about the Great > >>>>>> Renaming > >>>>>>>> in > >>>>>>>>> core > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably > >>> get > >>>>>> some > >>>>>>>>>>> learnings > >>>>>>>>>>>> from there? Who knows! > >>>>>>>>>>>> > >>>>>>>>>>>> Guido's blog (2009): > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>> > >>> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > >>>>>>>>>>> < > >>>>>>>>> > >>>>>>> > >>>>> > >>> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > >>>>>>>>> > >>>>>>>>>>>> Follow-up discussion (2024): > >>>>>>>>>>>> > >>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082 > >>>>>>> < > >>>>>>>>>>> > >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082> > >>>>>>>>>>>> > >>>>>>>>>>>> Cheers, > >>>>>>>>>>>> Omkar > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < > >>> weilee...@gmail.com > >>>>>>>>> <mailto: > >>>>>>>>>>> weilee...@gmail.com>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I think we should probably just accept it as an airflow > >>>>> term. > >>>>>>> At > >>>>>>>>> least, > >>>>>>>>>>>>> that’s how I understood it when I first used Airflow. I > >>>>> feel > >>>>>>>>> renaming > >>>>>>>>>>> it > >>>>>>>>>>>> at > >>>>>>>>>>>>> this stage would require considerable effort from > >>>>> maintainers > >>>>>>> and > >>>>>>>>>>>> existing > >>>>>>>>>>>>> users without providing equivalent benefits. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Best, > >>>>>>>>>>>>> Wei > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < > >>>>>> kaxiln...@gmail.com > >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle > >>>>> this > >>>>>>>>> discussion > >>>>>>>>>>>> for > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much > >>>>>>> disruption > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau > >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: > >>>>>>>>> consta...@astronomer.io.inva>lid> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow > >>>>>> experience > >>>>>>>>> what a > >>>>>>>>>>>> dag > >>>>>>>>>>>>> is, > >>>>>>>>>>>>>>> they'll start talking about workflow attributes - > >>> stuff > >>>>>> like > >>>>>>>>> dags > >>>>>>>>>>>> being > >>>>>>>>>>>>> a > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure > >>>>>> doesn't > >>>>>>>>> come up. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to > >>> embrace > >>>>> the > >>>>>>> name > >>>>>>>>> and > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: > >>>>> vik...@astronomer.io.inva > >>>>>>>>> lid> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember > >>>>> struggling > >>>>>>> with > >>>>>>>>> this > >>>>>>>>>>>>> when I > >>>>>>>>>>>>>>>> started working with Airflow. > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an > >>>>>>>> established > >>>>>>>>>>>> concept > >>>>>>>>>>>>>>> now > >>>>>>>>>>>>>>>> regardless of the origin. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective > >>> best > >>>>>>>> expressed > >>>>>>>>> by > >>>>>>>>>>>>> Daniel > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather > >>> than > >>>>>> the > >>>>>>>>> computer > >>>>>>>>>>>>>>> science > >>>>>>>>>>>>>>>> concept. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>> Vikram > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: > >>>>>> oniko...@amazon.com.inva > >>>>>>>>> lid > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're > >>> right > >>>>>> Ryan, > >>>>>>>> DAG > >>>>>>>>>>> isn't > >>>>>>>>>>>>>>> great > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will > >>> cause > >>>>> much > >>>>>>>> more > >>>>>>>>>>>> wreckage > >>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>> it solves. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from > >>>>> defining > >>>>>>>> DAG. > >>>>>>>>> I > >>>>>>>>>>>> think > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community > >>> for a > >>>>>> while > >>>>>>>> now > >>>>>>>>>>>> anyway, > >>>>>>>>>>>>>>> so > >>>>>>>>>>>>>>>>> that feels like a natural step. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>> Niko > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ________________________________ > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: > >>>>>>>>> a...@apache.org>> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > >>>>>> dev@airflow.apache.org > >>>>>>>> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the > >>> term > >>>>>> "DAG" > >>>>>>>>> for end > >>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the > >>>>>>>>> organization. > >>>>>>>>>>> Do > >>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>> click links or open attachments unless you can > >>> confirm > >>>>>> the > >>>>>>>>> sender > >>>>>>>>>>>> and > >>>>>>>>>>>>>>>> know > >>>>>>>>>>>>>>>>> the content is safe. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient > >>> d’un > >>>>>>>>> expéditeur > >>>>>>>>>>>>>>> externe. > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > >>>>>> jointe > >>>>>>> si > >>>>>>>>> vous > >>>>>>>>>>> ne > >>>>>>>>>>>>>>>> pouvez > >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si > >>> vous > >>>>>> n’êtes > >>>>>>>> pas > >>>>>>>>>>>> certain > >>>>>>>>>>>>>>>> que > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a > >>> term — > >>>>>>>> getting > >>>>>>>>> to > >>>>>>>>>>>>> re-use > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, both > >>>>> sides > >>>>>>> have > >>>>>>>>> good > >>>>>>>>>>>>>>> reasons > >>>>>>>>>>>>>>>>> presented. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -a > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish > >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes > >>> from, > >>>>>>> "well, > >>>>>>>>> no > >>>>>>>>>>> one > >>>>>>>>>>>>>>>>> actually > >>>>>>>>>>>>>>>>>> knows but..." and then make something up. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < > >>>>>>>>> ja...@potiuk.com > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is > >>> the > >>>>>>>>>>> tongue-twister, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < > >>>>>>>>> ja...@potiuk.com > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I > >>> would > >>>>> very > >>>>>>>> much > >>>>>>>>> want > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>> able > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it > >>> further. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk > >>> it's a > >>>>>>>>>>>> tongue-twister, > >>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to > >>> pronounce > >>>>>> it > >>>>>>>>>>> properly. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> J. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: > >>>>>>>> br...@astronomer.io.inva > >>>>>>>>>> lid> > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when > >>>>> starting > >>>>>> on > >>>>>>>>> AIP-38 > >>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>> modernize > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more > >>>>>>>>> descriptive of > >>>>>>>>>>>> what > >>>>>>>>>>>>>>>> one > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph > >>> is an > >>>>>>>>>>> implementation > >>>>>>>>>>>>>>>>> detail. > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this > >>> point > >>>>>> "DAG" > >>>>>>>> has > >>>>>>>>>>> become > >>>>>>>>>>>>>>>> "dag" > >>>>>>>>>>>>>>>>>>> , a > >>>>>>>>>>>>>>>>>>>>> word in its own right. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community > >>>>>> discussion, > >>>>>>>>> Airflow > >>>>>>>>>>>>>>>> Summit > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's > >>>>>> embrace > >>>>>>>>> "dag". > >>>>>>>>>>> A > >>>>>>>>>>>>>>> user > >>>>>>>>>>>>>>>>>>> just > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical > >>>>> concept > >>>>>>>>> behind > >>>>>>>>>>> that > >>>>>>>>>>>>>>>>> word. I > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than > >>> refactoring so > >>>>>>> much > >>>>>>>>> code, > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow > >>>>> questions, > >>>>>>>> etc. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel > >>> Standish > >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot > >>> of > >>>>>> pain > >>>>>>>> for > >>>>>>>>>>>>>>>>> questionable > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the > >>> idea. I > >>>>>>> agree > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> association > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less > >>>>>>> invasive > >>>>>>>>>>>>>>> mitigations. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to > >>>>>> de-emphasize > >>>>>>>> the > >>>>>>>>>>> math > >>>>>>>>>>>>>>> nerd > >>>>>>>>>>>>>>>>>>>>> origins > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / > >>> website / > >>>>>> etc, > >>>>>>>>> *never > >>>>>>>>>>>>>>>> define* > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic > >>>>>> graph. > >>>>>>>>> Always > >>>>>>>>>>>>>>> define > >>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>> as a > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is > >>> like a > >>>>>>>>> historical > >>>>>>>>>>>>>>>>> footnote, > >>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere > >>> hidden. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the > >>> docs > >>>>> in > >>>>>>>>> general > >>>>>>>>>>> e.g. > >>>>>>>>>>>>>>>>>>> writing > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / > >>> "DAGs" > >>>>>> etc. > >>>>>>>> The > >>>>>>>>>>>> upper > >>>>>>>>>>>>>>>> case > >>>>>>>>>>>>>>>>>>>>> part > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but > >>> "dag" in > >>>>>>>>> airlfow is > >>>>>>>>>>>> just > >>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with > >>> "DAGs" > >>>>> is > >>>>>>> not > >>>>>>>>> really > >>>>>>>>>>>>>>>>>>> unhelpful. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow > >>> is > >>>>> its > >>>>>>> own > >>>>>>>>>>> thing, > >>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> *not* strictly > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which > >>> nobody > >>>>>> knows > >>>>>>>>> about > >>>>>>>>>>>>>>>> anyway), > >>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that > >>> normal > >>>>>>> people > >>>>>>>>>>>>>>> understand. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk > >>> < > >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it > >>> will > >>>>> be > >>>>>>>>> extremely > >>>>>>>>>>>>>>>>>>>>> difficult to > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it > >>> will > >>>>>> make > >>>>>>> a > >>>>>>>>> lot of > >>>>>>>>>>>>>>>>>>> "google" > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not > >>>>> finding > >>>>>>> the > >>>>>>>>> right > >>>>>>>>>>>>>>>>>>> answers. > >>>>>>>>>>>>>>>>>>>>>> This > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - > >>> besides the > >>>>>>>>> community > >>>>>>>>>>> and > >>>>>>>>>>>>>>>> ideas > >>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of > >>>>> examples, > >>>>>>>>> problems > >>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>> solutions > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to > >>> remember > >>>>>> that > >>>>>>>>> all the > >>>>>>>>>>>> AI > >>>>>>>>>>>>>>>>>>>>> trained > >>>>>>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly > >>> matching > >>>>>>> queries > >>>>>>>> of > >>>>>>>>>>>> people. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily > >>>>>> switch. > >>>>>>>>> And if > >>>>>>>>>>> we > >>>>>>>>>>>>>>> do > >>>>>>>>>>>>>>>> - > >>>>>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline > >>> instead > >>>>> of > >>>>>>>>> `dag` if > >>>>>>>>>>>> not > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> above > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor > >>> that it > >>>>>>> might > >>>>>>>>> cause > >>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>>>>> problems > >>>>>>>>>>>>>>>>>>>>>>> than it solves. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will > >>> think > >>>>>>> it's > >>>>>>>> a > >>>>>>>>> good > >>>>>>>>>>>>>>>> idea, I > >>>>>>>>>>>>>>>>>>>>> am > >>>>>>>>>>>>>>>>>>>>>> ok > >>>>>>>>>>>>>>>>>>>>>>> with it. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> J, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek > >>> Bhakat > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva > >>> <mailto: > >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less > >>> sense > >>>>> to > >>>>>>>>> someone > >>>>>>>>>>> new > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> workflow > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the > >>> nature > >>>>>> of > >>>>>>>>> being > >>>>>>>>>>>>>>> acyclic. > >>>>>>>>>>>>>>>>>>>>> Sure, > >>>>>>>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to > >>> workaround it. > >>>>>>>> Still, > >>>>>>>>> in > >>>>>>>>>>> my > >>>>>>>>>>>>>>>>>>>>> opinion, > >>>>>>>>>>>>>>>>>>>>>>> there > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in > >>>>>>> workflow > >>>>>>>>>>>>>>>>>>> orchestration. > >>>>>>>>>>>>>>>>>>>>>> Most > >>>>>>>>>>>>>>>>>>>>>>>> (*if > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be > >>>>> covered > >>>>>>>>> using an > >>>>>>>>>>>>>>>> acyclic > >>>>>>>>>>>>>>>>>>>>>> manner > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the > >>> idempotency. So > >>>>> I > >>>>>>>> would > >>>>>>>>> want > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> "acyclic" > >>>>>>>>>>>>>>>>>>>>>>>> word to stick. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>>>>>> Avi > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: > >>> bernd.stroe...@kosakya.de>> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an > >>>>> Airflow > >>>>>>> Fan; > >>>>>>>>> so > >>>>>>>>>>> many > >>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>> ideas. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be > >>>>>> replaced > >>>>>>> by > >>>>>>>>> the > >>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>>>>> general > >>>>>>>>>>>>>>>>>>>>>>>> Term > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow > >>>>>> (Petri) > >>>>>>>> Net > >>>>>>>>>>> (AFN) > >>>>>>>>>>>>>>>>>>> (maybe > >>>>>>>>>>>>>>>>>>>>>>>> without > >>>>>>>>>>>>>>>>>>>>>>>>> a direction); > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a > >>>>> Graph > >>>>>>>>> Database. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow > >>> Graph > >>>>>> (AFG) > >>>>>>>>> might be > >>>>>>>>>>>>>>>>>>>>> assigned > >>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member > >>> of a > >>>>>>>> library. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different > >>>>> structure > >>>>>>>> than > >>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>> configuration > >>>>>>>>>>>>>>>>>>>>>>>>> Graph. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > >>>>>>>>>>> bernd.stroe...@gmail.com> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov > >>> <ikholo...@google.com.inva > >>>>>>>>> <mailto: > >>>>>>>>>>> ikholo...@google.com.inva>LID> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > >>>>>>>>> dev@airflow.apache.org > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the > >>> term > >>>>>>> "DAG" > >>>>>>>>> for > >>>>>>>>>>> end > >>>>>>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly > >>>>>> suboptimal, > >>>>>>> it > >>>>>>>>> is > >>>>>>>>>>> part > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>> Airflow > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many > >>> levels, > >>>>> that > >>>>>>> any > >>>>>>>>>>> attempt > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> change > >>>>>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not > >>> reduce it. > >>>>>> The > >>>>>>>>> only > >>>>>>>>>>>> thing > >>>>>>>>>>>>>>>>>>>>> that is > >>>>>>>>>>>>>>>>>>>>>>>> worse > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is > >>> when > >>>>>>> there > >>>>>>>>> are > >>>>>>>>>>> two > >>>>>>>>>>>>>>> ways > >>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>> define > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and > >>>>> tutorials > >>>>>>>> will > >>>>>>>>>>>> suddenly > >>>>>>>>>>>>>>>>>>>>> become > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows > >>> as > >>>>>>> "DAG"s. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the > >>> users > >>>>> away > >>>>>>>> with > >>>>>>>>> a > >>>>>>>>>>>> number > >>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>> breaking > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more > >>>>>> breaking > >>>>>>>>> changes > >>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> most > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people > >>> are > >>>>>>>> looking > >>>>>>>>> for. > >>>>>>>>>>>>>>>>>>>>> Attempting > >>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be > >>>>> interpreted > >>>>>>> as > >>>>>>>> an > >>>>>>>>>>> even > >>>>>>>>>>>>>>>>>>>>> stronger > >>>>>>>>>>>>>>>>>>>>>>>> signal > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing > >>> the > >>>>>> term > >>>>>>>> in > >>>>>>>>> the > >>>>>>>>>>>> long > >>>>>>>>>>>>>>>>>>>>> run. I > >>>>>>>>>>>>>>>>>>>>>>>> even > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating > >>> it in > >>>>>> the > >>>>>>>> DAG > >>>>>>>>>>>>>>> definition > >>>>>>>>>>>>>>>>>>>>>>>> interface. > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in > >>> other > >>>>>> places > >>>>>>>> in > >>>>>>>>>>>> Airflow, > >>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards, > >>>>>>>>>>>>>>>>>>>>>>>>> Igor > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas > >>> Harenslak > >>>>>>>>>>>>>>>>>>>>>>> <b...@astronomer.io.inva <mailto: > >>>>>>>> b...@astronomer.io.inva > >>>>>>>>>> lid > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of “DAG” > >>> have > >>>>>>> already > >>>>>>>>> faded > >>>>>>>>>>>> over > >>>>>>>>>>>>>>>>>>>>> time, > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to > >>>>>> create > >>>>>>>>> cyclic > >>>>>>>>>>>>>>>>>>> graphs, > >>>>>>>>>>>>>>>>>>>>>> e.g. > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine > >>>>> these > >>>>>>>>>>> properties > >>>>>>>>>>>>>>>>>>>>>> vanishing > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that > >>>>>>> perspective I > >>>>>>>>>>> support > >>>>>>>>>>>>>>>>>>>>>> changing > >>>>>>>>>>>>>>>>>>>>>>>>>> “DAG" to a more generic name. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do > >>>>>> naming: > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think “workflow” is the most fitting > >>> name. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this > >>> change, I > >>>>>>> suggest > >>>>>>>>>>>> defining > >>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>> clear > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce > >>> the > >>>>>>>>> deprecation in > >>>>>>>>>>>>>>>>>>>>> Airflow > >>>>>>>>>>>>>>>>>>>>>> 3, > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove “DAG” in Airflow 4? > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < > >>>>>>>> neil4r...@gmail.com > >>>>>>>>>>> <mailto:neil4r...@gmail.com>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG, > >>>>>>>> especially > >>>>>>>>> when > >>>>>>>>>>>>>>>>>>> most > >>>>>>>>>>>>>>>>>>>>>> other > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term > >>> wholeheartedly. > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or > >>>>> confusing > >>>>>>>>> about it > >>>>>>>>>>> at > >>>>>>>>>>>>>>>>>>>>> all, > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly > >>>>>>> confusing > >>>>>>>>> to > >>>>>>>>>>> most > >>>>>>>>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>>>>>>>> who > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been > >>>>>>>>>>>>>>>>>>>>>>>>>> using > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping > >>>>> Chung > >>>>>>>>>>>>>>>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: > >>>>>>>>> t...@astronomer.io.inva > >>>>>>>>>>>> lid > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with > >>> the > >>>>> term > >>>>>>>> DAG. > >>>>>>>>> The > >>>>>>>>>>>> only > >>>>>>>>>>>>>>>>>>>>>>> problem > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling people—including > >>>>>>> myself—to > >>>>>>>>> stop > >>>>>>>>>>>> using > >>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term) > >>>>>>>>>>>>>>>>>>>>>>>>>> is to > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can’t recall who, but someone > >>> mentioned > >>>>>>>>> “workflow” is > >>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>> very > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term “definition” was > >>>>>>> proposed, > >>>>>>>>> but > >>>>>>>>>>>> it’s a > >>>>>>>>>>>>>>>>>>>>> bit > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few > >>> places > >>>>> and > >>>>>>> kept > >>>>>>>>>>> finding > >>>>>>>>>>>>>>>>>>>>> myself > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting “what definition?” and > >>> wanting to > >>>>>>>> clarify > >>>>>>>>> “DAG > >>>>>>>>>>>>>>>>>>>>>>> definition” > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose). > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens > >>> Scheffler > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > >>>>>>>>>>> j_scheff...@gmx.de.inva>LID> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the > >>> exactly > >>>>> same > >>>>>>>>>>>> observation, > >>>>>>>>>>>>>>>>>>>>> had a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is > >>> always > >>>>> an > >>>>>>>>>>> introduction > >>>>>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone > >>>>>>>>>>>>>>>>>>>>>>>>>> joins > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming > >>> makes > >>>>>>> sense. > >>>>>>>>>>>>>>>>>>> Especially > >>>>>>>>>>>>>>>>>>>>>> when > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a > >>>>>>> reasonable > >>>>>>>>> step. > >>>>>>>>>>>>>>>>>>>>> Concepts > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding > >>>>> below > >>>>>>> the > >>>>>>>>> desk > >>>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 > >>>>>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we > >>> keep > >>>>>>>> details > >>>>>>>>> of > >>>>>>>>>>>>>>>>>>> python > >>>>>>>>>>>>>>>>>>>>>> names > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a > >>> DAG… > >>>>>> but > >>>>>>>> user > >>>>>>>>>>>> facing > >>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>> is a > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan > >>> Hatter < > >>>>>>>>>>>>>>>>>>>>>>> ryan.hat...@astronomer.io <mailto: > >>>>>>>>> ryan.hat...@astronomer.io > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your > >>> swords... at > >>>>>>> least > >>>>>>>>> for > >>>>>>>>>>>> now. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little > >>> meaning to > >>>>>>>> Airflow > >>>>>>>>>>>> users. > >>>>>>>>>>>>>>>>>>>>>> Indeed, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>>> has > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some > >>>>>> mathematicians > >>>>>>>> and > >>>>>>>>>>>>>>>>>>> software > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers > >>>>>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually > >>>>>> matter. > >>>>>>>> For > >>>>>>>>>>>> someone > >>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow > >>>>> orchestration, > >>>>>>> one > >>>>>>>>> of > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> first > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a > >>> DAG?" > >>>>>> The > >>>>>>>>> answer > >>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>> almost > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic > >>> graph. > >>>>> You > >>>>>>>> don't > >>>>>>>>>>> need > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> worry > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what > >>>>>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your > >>>>> workflow." > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at > >>>>> least > >>>>>> a > >>>>>>>>> couple > >>>>>>>>>>>>>>>>>>>>> important > >>>>>>>>>>>>>>>>>>>>>>>>> reasons: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As > >>> mentioned > >>>>>>> above, > >>>>>>>>> "DAG" > >>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and > >>> confusing. > >>>>>> We > >>>>>>>> want > >>>>>>>>>>>> Airflow > >>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right > >>> off the > >>>>>> bat > >>>>>>>>> creates > >>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>> initial > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow > >>>>>>> Concepts*: > >>>>>>>>> The > >>>>>>>>>>> DAG > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>> just > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow > >>> workflow. The > >>>>>>>>> workflow > >>>>>>>>>>>>>>>>>>> includes > >>>>>>>>>>>>>>>>>>>>>> its > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen > >>>>> other > >>>>>>>>>>> parameters, > >>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> other > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that > >>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesn’t account for. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the > >>> Airflow > >>>>>>>> homepage > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > >>>>>>>>>>> https://airflow.apache.org/>>. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Airflow® is a platform > >>> created by > >>>>>> the > >>>>>>>>>>> community > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor > >>> workflows. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is > >>>>> Airflow?" > >>>>>>> docs > >>>>>>>>> page > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html < > >>>>>>>>>>> > >>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , > >>>>>>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what > >>> Airflow is > >>>>>>>> without > >>>>>>>>>>> using > >>>>>>>>>>>>>>>>>>>>> "DAG." > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the > >>> term > >>>>> is > >>>>>>>>>>> introduced > >>>>>>>>>>>>>>>>>>> out > >>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to > >>> explain > >>>>> it: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a > >>>>> collection > >>>>>>> of > >>>>>>>>> tasks > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs > >>> in > >>>>>> these > >>>>>>>>>>>>>>>>>>> introductions > >>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't > >>>>>> orchestrate > >>>>>>>>> DAGs; > >>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons > >>>>>> irrelevant > >>>>>>>> to > >>>>>>>>>>> almost > >>>>>>>>>>>>>>>>>>>>> every > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an > >>> alias > >>>>> for > >>>>>>> the > >>>>>>>>> term > >>>>>>>>>>>> "DAG" > >>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace > >>> "DAG" > >>>>>> with > >>>>>>>>>>>> "workflow". > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag( > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow( > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running > >>>>> away. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>> ------------------------------------------------------------------ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > >>>>>>>>>>>>>>>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > >>>>>>>>>>> dev-unsubscr...@airflow.apache.org> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: > >>>>>>>>>>>>>>>>>>>>> dev-h...@airflow.apache.org <mailto: > >>>>>>>>>>> dev-h...@airflow.apache.org> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
