Am intrigued. Curious about dynamic dag pattern, where you create the DAG object in a a create_dag function and adding the DAG to globals. Would this new way prevent someone from modifying the dag object within the function, or returning it?
> On Apr 27, 2022, at 11:20 AM, Ferruzzi, Dennis <ferru...@amazon.com.INVALID> > wrote: > > I don't know what it would take under the hood, but I'm intrigued. From a > user perspective, anything to make the DAG DRYer is a win IMHO. > > ________________________________________ > From: Malthe <mbo...@gmail.com> > Sent: Wednesday, April 27, 2022 1:49 AM > To: dev@airflow.apache.org > Subject: [EXTERNAL] Implicit DAG registration > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > DAGs must be registered at the module top-level. During dagbag > processing we have: > > top_level_dags = ((o, m) for m in mods for o in > m.__dict__.values() if isinstance(o, DAG)) > > It makes sense of course – we can't have DAGs floating in space, they > need an anchor. > > Or do they? > > It would be entirely possible for the DAG constructor to register > itself with some global registry, obviating for the follow pattern: > > with DAG(...) as dag: > ... > > I think most users would prefer: > > with DAG(...): > ... > > Since there is generally speaking no need for the alias. > > Downsides? You have less control of what DAGs are made available – but > really, does it make sense to create a DAG object only to drop it > immediately after? > > I volunteer to implement this if there's a positive feedback. > > Cheers
