That sounds like a good idea at first, but falls down with possible code changes in operators between one task and the next.
(I would like this, but there are definite complexities) -ash On 31 January 2019 16:56:54 GMT, Dan Davydov <ddavy...@twitter.com.INVALID> wrote: >I feel the right higher-level solution to this problem (which is >"Adding >Consistency to Airflow") is DAG serialization, that is all DAGs should >be >represented as e.g. JSON (similar to the current SimpleDAGBag object >used >by the Scheduler). This solves the webserver issue, and also adds >consistency between Scheduler/Workers (all DAGruns can be ensured to >run at >the same version of a DAG instead of whatever happens to live on the >worker >at the time). > >On Thu, Jan 31, 2019 at 9:44 AM Peter van ‘t Hof < >petervant...@godatadriven.com> wrote: > >> Hi All, >> >> As most of you guys know, airflow got an issue when loading new dags >where >> the webserver sometimes sees it and sometimes not. >> Because of this we did wrote this AIP to solve this issue: >> >> >https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB >> >> Any feedback is welcome. >> >> Gr, >> Peter van 't Hof >> Big Data Engineer >> >> GoDataDriven >> Wibautstraat 202 >> 1091 GS Amsterdam >> https://godatadriven.com >> >>
