I feel the right higher-level solution to this problem (which is "Adding Consistency to Airflow") is DAG serialization, that is all DAGs should be represented as e.g. JSON (similar to the current SimpleDAGBag object used by the Scheduler). This solves the webserver issue, and also adds consistency between Scheduler/Workers (all DAGruns can be ensured to run at the same version of a DAG instead of whatever happens to live on the worker at the time).
On Thu, Jan 31, 2019 at 9:44 AM Peter van ‘t Hof < petervant...@godatadriven.com> wrote: > Hi All, > > As most of you guys know, airflow got an issue when loading new dags where > the webserver sometimes sees it and sometimes not. > Because of this we did wrote this AIP to solve this issue: > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB > > Any feedback is welcome. > > Gr, > Peter van 't Hof > Big Data Engineer > > GoDataDriven > Wibautstraat 202 > 1091 GS Amsterdam > https://godatadriven.com > >
