The key thing that isn't possible is the automatic clearing. If I notice a problem in the data after the cluster has been torn down and I want to the rerun a single task then you have to manually clear the create cluster task and the task you want to run AND the teardown task. (This is not a hypothetical but a real issue I have had)
More generally this is part of the trend of giving DAG authors the ability to specify "directly" the constructs and flows that they currently have to hack around. -a On 15 January 2023 00:26:31 GMT, Alexander Shorin <kxe...@gmail.com> wrote: >Hi! > >This looks like sugar without any practical difference from explicitly >defined setup/teardown operators in dag. And the example doesn't add any >automatic'sation except to set you free from defining explicit relations >between setup, teardown and regular operators, but why all of this when you >already could do this explicitly and reliable? Looks like a feature just >for a feature. > >If that example of implementation could hide within a context details about >ti.xcom_pull(task_id="create_cluster") and hide the need of explicit call >setup/teardown operators - that would be a bit helpful, but there is no >such a thing. > >-- >,,,^..^,,, > > >On Mon, Jan 9, 2023 at 7:27 PM Ash Berlin-Taylor <a...@apache.org> wrote: > >> Hello everyone, >> >> I am calling for a vote on AIP-52 >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-52+Automatic+setup+and+teardown+tasks >> There haven't been any notable changes to the original document, mostly >> just clairifications to the proposal. >> >> This is my +1, and the vote will last until 10am (UTC) on Monday 16th >> January. >> >> Thanks, >> Ash >> (To remind people what this is about, here is the example from the first >> dicuss email) >> >> ``` >> from airflow import DAG, task, setup, teardown >> >> >> with DAG(dag_id='test'): >> @setup >> def create_cluster(): >> ... >> return cluster_id >> >> @task >> def load(ti): >> # Example: >> cluster_id = ti.xcom_pull(task_id="create_cluster") >> >> def summarize(): >> ... >> >> @teardown(on_failure_fail_dagrun=False) >> def teardown_cluster(): >> ... >> cluster_id = ti.xcom_pull(task_id="create_cluster") >> >> create_cluster() >> load() >> summarize() >> teardown_cluster() >> ``` >> >>
