I really like that - in my opinion definitely worth a note in the docs. Let's just make sure we warn users of side effects which you already mentioned here.
From what I can say about dynamic dag generation at Trade Republic is that it was a huge pain for us to get it done right. So that it does not negatively impact how our airflow instance performs. Just one poorly written dymanic dag generator can cause huge damage to airflows overall healthiness. Best, Felix Sent with [Proton Mail](https://proton.me/) secure email. ------- Original Message ------- On Monday, July 18th, 2022 at 11:34, Jarek Potiuk <ja...@potiuk.com> wrote: > I've added a PR to our docs (howto/dynamic-dags) to guide our users into this > possibility. > > https://github.com/apache/airflow/pull/25121 > > On Tue, Jul 12, 2022 at 11:25 AM Abhishek Bhakat > <abhishek.bha...@astronomer.io.invalid> wrote: > >> Can vote for making it as an optional approach for fine-tuning (only for >> advance users). >> >> On 12-Jul-2022 at 7:44:35 AM, Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> Not interesting :) ? >>> >>> On Thu, Jul 7, 2022 at 10:41 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> >>> >>>> Hello everyone, >>> >>>> >>> >>>> We have just published a blog on our medium - >>>> https://medium.com/apache-airflow/airflows-magic-loop-ec424b05b629 - that >>>> is a blog of one of our users Itay Bittan (thanks!) who had been inspired >>>> by our discussion on Slack on how they struggle with delays of loading >>>> dynamic dags in their K8S. >>> >>>> >>> >>>> The problem that they had was that they have dynamic dags that are created >>>> in a big loop (1000s of DAGs) and that caused ~ 2 minutes delays on >>>> starting their tas on K8S, because all DAGs have to be created by the loop. >>> >>>> >>> >>>> What I proposed to try (since the DAGs were connected by the loop but >>>> really isolated from each other) is to skip "all other" DAG creation in >>>> the loop when it is parsed in the worker. That resulted in cutting the >>>> delay to ~ 200ms. >>> >>>> >>> >>>> His case seems to be general enough to maybe suggest it even as a >>>> "general" solution - currently it is based on possibly several >>>> "non-documented" assumptions (that dag_id is passed in a certain way to >>>> the worker and that you can use it to filter out such a loop. >>> >>>> >>> >>>> However maybe that's a good idea to make it documented and convert into >>>> "best practice" when you have similar Dynamic DAGs. >>> >>>> >>> >>>> I can think of several caveats of such an approach - not all DAGs created >>>> in a loop can be isolated, sometimes there might be side-effects that make >>>> your dag have different structure if you skip other DAGs, but - I thought >>>> that if we add some "guidelines" that could be easily replicated by other >>>> users. >>> >>>> >>> >>>> WDYT? >>> >>>> >>> >>>> J.
