Great stuff!! Thanks to everyone who contributed 👏 On Sat, Mar 14, 2020 at 2:26 PM Deng Xiaodong <xd.den...@gmail.com> wrote:
> It is really great news and would be a big plus. > > Thanks Kaxil, Ash, Anita, and Zhou Fang! > > > XD > > > On 14 Mar 2020, at 2:31 AM, Kevin Yang <yrql...@gmail.com> wrote: > > > > This is thrilling news! Completely upleveling the experience using and > > maintaining the webserver. Thank you so much everyone who contributed to > > this initiative! > > > > > > Cheers, > > Kevin Y > > > > On Fri, Mar 13, 2020 at 6:24 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > >> Hi all, > >> > >> Happy to tell you all that we have completed the first phase of DAG > >> Serialisation i.e. the Webserver is stateless and can now run without > >> access to DAG Files. > >> > >> The 2 limitations we had in 1.10.7-1.10.9 ( > >> > https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations) > >> have been resolved. > >> > >> Special thanks to @ash for his continuous guidance and contributions. > >> > >> Also a special mention to Anita Fronczak and Zhou Fang for their > >> contributions along the way. > >> > >> The next step is to remove SimpleDag representation in the Scheduler and > >> replace it with Serialized DAG (WIP PR: > >> https://github.com/apache/airflow/pull/7694) > >> > >> *Advantages*: > >> > >> - *Reduction in Webserver startup time* for large number of DAGs. > >> Without DAG Serialization all the DAGs are loaded in the DagBag during > >> the > >> Webserver startup. With DAG Serialization, an empty DagBag is created > >> and > >> Dags are loaded from DB only when needed (i.e. when a particular DAG > is > >> clicked on in the home page) > >> - *No DAG Parsing / Consistency*: Webserver would load DAGs from DB > and > >> won't even need the DAG Files when DAG Serialization is turned on. > DAGs > >> are > >> parsed, serialized and stored in DB by the Scheduler. > >> - Rendered Templates for TasksInstances that have already run will now > >> correctly display their value which was true at the time of the run > >> instead > >> of the current value. > >> - Paves way for* DAG Versioning* (more details on it when I create a > >> separate AIP / update an existing AIP for it) and *Scheduler HA > *(AIP-15 > >> < > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 > >>> > >> ). > >> > >> > >> I will create new JIRA issues for further steps with DAG Serialization > and > >> DAG Versioning and would discuss them in our next sig-dag-serialization > >> call (later this month). > >> > >> Regards, > >> Kaxil > >> > >
