Hi everyone, I'd like to start a discussion about a new feature we'd like to add to Airflow we call "setup and tear down tasks"
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks Lets start with a code example, that if you are familiar with unittest.TestCase I hope will be familiar and you can guess what the effect would be: ``` from airflow import DAG, task, setup, teardown with DAG(dag_id='test'): @setup def create_cluster(): ... return cluster_id @task def load(ti): # Example: cluster_id = ti.xcom_pull(task_id="create_cluster") def summarize(): ... @teardown(on_failure_fail_dagrun=False) def teardown_cluster(): ... cluster_id = ti.xcom_pull(task_id="create_cluster") create_cluster() load() >> summarize() teardown_cluster() ``` (This has been an itch of mine that I've had since I first started using Airflow back in 2017!) We go in to quite a bit of detail about the semantics and behaviours of these new task types, but the tl;dr (copied from the AIP doc) Add a new syntax for marking tasks as setup/teardown that: * Teardown tasks will still run even if the upstream tasks have otherwise failed * Teardown tasks failing don't always result in the DagRun being marked as failed (up to DAG author to choose failure mode) * Automatically clear setup/teardown tasks when clearing a dependent task I'm not sure when we'll start a vote on this due to the holiday season, but we'd like to start working on this in January. -ash
