Hey folks, Quick update on the trigger migration work tracked in #65733, all the provider PRs have been merged.
- EMR Serverless (#65740) — thanks @riyanmohmmeed-dev - KubernetesPodTrigger (#65741) — thanks @raushanprabhakar1 - Dataproc (#65742) — thanks @srchilukoori Really appreciate folks picking these up so quickly after the issue went out. The on_kill() hook on BaseTrigger now has real users across AWS, GCP, and Kubernetes — exactly the kind of breadth I was hoping for. I think that there's a long tail of triggers that could benefit from this utility like the AzureDataFactoryTrigger, DbtCloudRunJobTrigger, and some more. These seem to have the intent, and can benefit from the hook method. If you think you can pick up any of these, feel free to PR it, and tag me for a review. Thanks & Regards, Amogh Desai On Fri, Apr 24, 2026 at 11:53 AM Amogh Desai <[email protected]> wrote: > Thanks all. > > Jens - I mentioned it above: > > > If you are someone building or maintaining a trigger that manages > external work, you can consider adopting to this hook > and implementing it for your trigger class, Airflow 3.3 will launch this. > > Jason - yep, I can imagine that pain :) > > Thanks & Regards, > Amogh Desai > > > On Fri, Apr 24, 2026 at 3:23 AM Zhe-You(Jason) Liu <[email protected]> > wrote: > >> Thanks Amogh for adding the `on_kill` method to `BaseTrigger`! >> >> Without your change, the only way for me to achieve an explicit kill from >> the CLI or UI is by adding a `@teardown` task at the end of the external >> job submission for the "sentinel" purpose. >> Your change even makes this better, we can achieve the explicit kill with >> an atomic task! >> >> Best, >> Jason >> >> On Fri, Apr 24, 2026 at 4:14 AM Jens Scheffler <[email protected]> >> wrote: >> >> > Thanks Amogh for adding this! >> > >> > In deed this was missing for a long time! >> > >> > You did not tell about but we can assume plan is to release this with >> > Airflow 3.3.0? >> > >> > On 23.04.26 18:03, Aritra Basu wrote: >> > > This is quite useful! Nice! >> > > Thanks and Regards, >> > > Aritra Basu >> > > >> > > >> > > On Thu, Apr 23, 2026 at 9:17 PM Amogh Desai <[email protected]> >> > wrote: >> > > >> > >> Hi Again, >> > >> >> > >> Created an issue tracking the migration of provider triggers from the >> > old >> > >> CancelledError >> > >> pattern to the new on_kill() hook: >> > >> https://github.com/apache/airflow/issues/65733 >> > >> >> > >> Three triggers need updating — Dataproc, EMR Serverless, and >> Kubernetes. >> > >> Each is a >> > >> contained change in a single file. Good first contribution if you >> want >> > to >> > >> get familiar with the >> > >> triggerer. >> > >> >> > >> Drop a comment on the issue if you want to pick one up. >> > >> >> > >> Thanks & Regards, >> > >> Amogh Desai >> > >> >> > >> >> > >> On Thu, Apr 23, 2026 at 7:40 PM Kaxil Naik <[email protected]> >> wrote: >> > >> >> > >>> Finally, this has been troubling users since Triggerer was added. >> > >>> >> > >>> On Thu, 23 Apr 2026 at 14:59, Elad Kalif <[email protected]> >> wrote: >> > >>> >> > >>>> This is great! well done! >> > >>>> >> > >>>> On Thu, Apr 23, 2026 at 4:42 PM Amogh Desai <[email protected] >> > >> > >>> wrote: >> > >>>>> Hey All, >> > >>>>> >> > >>>>> Wanted to call out a new hook we added to BaseTrigger — >> `on_kill()`, >> > >>> PR: >> > >>>>> https://github.com/apache/airflow/pull/65590. >> > >>>>> >> > >>>>> *Why does it exist?* >> > >>>>> >> > >>>>> If your trigger polls an external job (BigQuery, Databricks, >> > >> Dataproc, >> > >>>>> etc.), there was no clean way to cancel that job when >> > >>>>> a user kills the deferred task, either by marking it as success, >> > >>> failure, >> > >>>>> or clearing it. `cleanup()` wasn't the right place — >> > >>>>> it fires on every trigger exit including triggerer restarts and >> > >> rolling >> > >>>>> deploys, so putting cancellation there would kill in-flight >> > >>>>> work on every redeploy. >> > >>>>> >> > >>>>> `on_kill()` solves this. It only fires when a user explicitly >> acts on >> > >>> the >> > >>>>> task. Not on restart, not on redistribution. >> > >>>>> >> > >>>>> *How it works* >> > >>>>> >> > >>>>> The triggerer passes a private sentinel via >> > >> `asyncio.Task.cancel(msg)` >> > >>> to >> > >>>>> distinguish user kills from other exits — no DB >> > >>>>> round-trip needed. There's a configurable timeout (*[triggerer] >> > >>>>> on_kill_timeout, default 30s*) so a slow external API call >> > >>>>> cannot block the triggerer. >> > >>>>> >> > >>>>> Databricks is updated as a reference: >> > >>>>> https://github.com/apache/airflow/pull/65672 >> > >>>>> >> > >>>>> Both `DatabricksExecutionTrigger` and >> > >>>>> `DatabricksSQLStatementExecutionTrigger` now implement it. >> > >>>>> >> > >>>>> *Who benefits from this?* >> > >>>>> >> > >>>>> If you are someone building or maintaining a trigger that manages >> > >>>> external >> > >>>>> work, you can consider adopting to this hook >> > >>>>> and implementing it for your trigger class, Airflow 3.3 will >> launch >> > >>> this. >> > >>>>> No compat guard is needed, Airflow < 3.2 the method just never >> gets >> > >>>> called. >> > >>>>> *Next steps* >> > >>>>> >> > >>>>> I pulled some data from the repo and these are candidates for this >> > >>>>> migration. >> > >>>>> >> > >>>>> Google Provider: DataprocSubmitTrigger and >> > >>> DataprocSubmitJobDirectTrigger >> > >>>>> in the google provider. Currently catches >> > >>>>> CancelledError in run() and calls hook.cancel_job(job_id) after >> > >>> checking >> > >>>>> safe_to_cancel(). >> > >>>>> >> > >>>>> Amazon Provider: EmrServerlessStartJobTrigger. Same pattern. >> > >>>> CancelledError >> > >>>>> handler with safe_to_cancel() calling >> > >>>>> hook.conn.cancel_job_run(). Straightforward migration. >> > >>>>> >> > >>>>> CNCF provider: KubernetesPodTrigger in cncf/kubernetes. Has an >> > >>>>> on_kill_action parameter controlling whether to >> > >>>>> delete/patch the pod. The cancellation logic lives in a >> > >> CancelledError >> > >>>>> handler. Move it to on_kill(). >> > >>>>> >> > >>>>> Happy to help with reviews for these. >> > >>>>> >> > >>>>> >> > >>>>> Thanks & Regards, >> > >>>>> Amogh Desai >> > >>>>> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> > >> >
