To me Airflow is not "open and play". It is not a closed system that guides you how to develop. Workflow as code requires more skills. There are stuff to learn before authoring dags. This discussion is about Variables but I might ask similar question about users who query API as part of top level code. This is also very bad and not covered by current discussion.
In my prespective users who hit this issue are likely also to hit many other issues. Its a good sign that they should try to learn Airflow better before continue development. I don't think a solution involves in hinding the issue is a good one. For now I'm -1 for making complecated changes to mitigate this. However I am infavor of finding a way to alert users that we detected a possible bad practice and we advise to recheck the code (that also requires some thought.. I don't know if bumping many warnings in the main UI is the right way...) בתאריך יום ו׳, 24 במרץ 2023, 00:25, מאת Vandon, Raphael <vand...@amazon.com.invalid>: > Hello, > I’d like to submit to discussion the idea of having a cache on Airflow > Variables. > The aim is to reduce DAG parsing time and Secret Manager API bill for > users who make a liberal use of Variable.get in their DAG files. > The drawback, of course, is that caches introduce a delay in how fast > changes are propagated. > > For improved readability, I kindly invite you to read and comment in the > github discussion : https://github.com/apache/airflow/discussions/30265 > > Cheers > >
