I'd prefer a world without separate pre_execute and post_execute functions
- as pointed out in the PR, they make reasoning about DAGs more complex,
and can be error prone.
Having said that, I know there are multiple users relying on these
functionalities, so I'll bring up my usual - another breaking change -
another obstacle to the AF3 adoption argument.
And as for relying on operators vs. PythonOperator + hooks - there are good
arguments for continuing relying on operators, or even rely on them more,
depending on customers' need and organizational setup.
On Fri, Mar 28, 2025 at 3:42 PM Tzu-ping Chung <t...@astronomer.io.invalid>
wrote:
> Passing post_execute as an argument is somewhat useful for operators that
> don’t support assets natively (most of them) but when you want to emit a
> dynamic path. For example:
>
> def _send_asset_event(context, result):
> # Rendered value!
> name = context["task"].output
> # Trigger an event against the emitted path.
> context["outlet_events"][write_data_outlet].add(Asset(name))
>
> write_data_outlet = AssetAlias("data")
>
> WriteSomeDataOperator(
> task_id="write_data",
> output="write_data_{{ run_id }}.parquet",
> outlets=[write_data_outlet],
> post_execute=_send_asset_event,
> )
>
> Without the functionality, you’ll have to write a subclass for each
> operator you want to do this, which is quite a bit boilerplate.
>
> Arguably this is only needed since we use operators too much. This
> wouldn’t be an issue if we rely more on the PythonOperator+hooks approach
> (like Bolke discussed at last year’s Airflow Summit), but alas, people
> don’t like to change how they do things, and operators are still very
> popular.
>
> The pre_execute argument *might* also be useful if you want to pre-process
> some values. That’s probably a lot less common, so I wouldn’t fret too much
> if it goes away. However, since post_execute and pre_execute basically use
> the same implementation, just one run right before and one right after
> execute, they should probably stay or go together.
>
> I think the ability of overriding pre_execute and post_execute in a
> subclass can definitely go away. They are practically useles; you can just
> put everything in execute, which always needs to exist in a BaseOperator
> subclass anyway.
>
> TP
>
>
> > On 28 Mar 2025, at 22:12, Kaxil Naik <kaxiln...@gmail.com> wrote:
> >
> > I am in favor of dropping support as they essentially do the same -- and
> > setup & teardown is more "native" (first-class UI support)
> >
> > On Fri, 28 Mar 2025 at 19:41, Kaxil Naik <kaxiln...@gmail.com> wrote:
> >
> >> Hi team,
> >>
> >> Should we drop support for pre_execute and post_execute for AF 3.0? They
> >> are still marked as experimental [1]. They were added [2] in a world
> >> without Setup and Teardown tasks.
> >>
> >> Regards,
> >>
> >> Kaxil
> >>
> >>
> >> [1]:
> >>
> https://github.com/apache/airflow/blob/7af0319ba16749f4aea78085dfe7823f321d262a/task-sdk/src/airflow/sdk/bases/baseoperator.py#L715-L724
> >> [2]: https://github.com/apache/airflow/pull/17576
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
> For additional commands, e-mail: dev-h...@airflow.apache.org
>
>
