Thanks Amogh for adding this!
In deed this was missing for a long time!
You did not tell about but we can assume plan is to release this with
Airflow 3.3.0?
On 23.04.26 18:03, Aritra Basu wrote:
This is quite useful! Nice!
Thanks and Regards,
Aritra Basu
On Thu, Apr 23, 2026 at 9:17 PM Amogh Desai <[email protected]> wrote:
Hi Again,
Created an issue tracking the migration of provider triggers from the old
CancelledError
pattern to the new on_kill() hook:
https://github.com/apache/airflow/issues/65733
Three triggers need updating — Dataproc, EMR Serverless, and Kubernetes.
Each is a
contained change in a single file. Good first contribution if you want to
get familiar with the
triggerer.
Drop a comment on the issue if you want to pick one up.
Thanks & Regards,
Amogh Desai
On Thu, Apr 23, 2026 at 7:40 PM Kaxil Naik <[email protected]> wrote:
Finally, this has been troubling users since Triggerer was added.
On Thu, 23 Apr 2026 at 14:59, Elad Kalif <[email protected]> wrote:
This is great! well done!
On Thu, Apr 23, 2026 at 4:42 PM Amogh Desai <[email protected]>
wrote:
Hey All,
Wanted to call out a new hook we added to BaseTrigger — `on_kill()`,
PR:
https://github.com/apache/airflow/pull/65590.
*Why does it exist?*
If your trigger polls an external job (BigQuery, Databricks,
Dataproc,
etc.), there was no clean way to cancel that job when
a user kills the deferred task, either by marking it as success,
failure,
or clearing it. `cleanup()` wasn't the right place —
it fires on every trigger exit including triggerer restarts and
rolling
deploys, so putting cancellation there would kill in-flight
work on every redeploy.
`on_kill()` solves this. It only fires when a user explicitly acts on
the
task. Not on restart, not on redistribution.
*How it works*
The triggerer passes a private sentinel via
`asyncio.Task.cancel(msg)`
to
distinguish user kills from other exits — no DB
round-trip needed. There's a configurable timeout (*[triggerer]
on_kill_timeout, default 30s*) so a slow external API call
cannot block the triggerer.
Databricks is updated as a reference:
https://github.com/apache/airflow/pull/65672
Both `DatabricksExecutionTrigger` and
`DatabricksSQLStatementExecutionTrigger` now implement it.
*Who benefits from this?*
If you are someone building or maintaining a trigger that manages
external
work, you can consider adopting to this hook
and implementing it for your trigger class, Airflow 3.3 will launch
this.
No compat guard is needed, Airflow < 3.2 the method just never gets
called.
*Next steps*
I pulled some data from the repo and these are candidates for this
migration.
Google Provider: DataprocSubmitTrigger and
DataprocSubmitJobDirectTrigger
in the google provider. Currently catches
CancelledError in run() and calls hook.cancel_job(job_id) after
checking
safe_to_cancel().
Amazon Provider: EmrServerlessStartJobTrigger. Same pattern.
CancelledError
handler with safe_to_cancel() calling
hook.conn.cancel_job_run(). Straightforward migration.
CNCF provider: KubernetesPodTrigger in cncf/kubernetes. Has an
on_kill_action parameter controlling whether to
delete/patch the pod. The cancellation logic lives in a
CancelledError
handler. Move it to on_kill().
Happy to help with reviews for these.
Thanks & Regards,
Amogh Desai
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
