This kind of ties back to Bolke’s Operator Must Die manifesto—you shouldn’t use the operator class here at all in this situation, but the corresponding hook instead.
Regarding preventing this anti-pattern, I can think of two ways: Add a check in BaseOperator to detect whether __init__ is called in a task work context, and error out guiding users to use hooks instead. Accept this as a valid pattern, and actually push to XCom. Personally I like options 1 more. Option 2 is admittedly less disruptive, but I’m pretty confident people will find other ways to use operators that do not work. Operators will never die, but they should fade away. <https://en.wikipedia.org/wiki/Old_soldiers_never_die> TP > On 27 Feb 2024, at 15:09, Jarek Potiuk <ja...@potiuk.com> wrote: > > Hello here, > > I have seen recently at least a few times that our users started to use a > strange pattern > > @task(task_id='some_id', provide_context=True) > def some_dummy_task(**context): > ti = context['ti'] > cmd2 = 'echo "pushing 2"' > dbt_task = BashOperator(task_id='some_dummy_task_id', > bash_command=cmd2, > do_xcom_push=True) > dbt_task.execute(context)` > > This **works** for them but then - for example - they complain that xcom is > not pushed (well of course it's not if you use operator and manually run > execute). > > Now - while this is a bad pattern - obviously - it seems that our users > **think** they can do itl. And maybe we can do something to prevent them > shooting themselves in their foot? > > I do not know that well the @task decoration magic, but maybe we could > somehow detect the case where someone instantiates the operator (and runs > execute) inside a decorated task and give a helpful error in this case? I > am afraid people will start using it more and more and the sooner we add > protection against it, the better chance we have to contain it. > > J.
