+1 to what Kaxil wrote.. On Thu, Oct 15, 2026 at 11:59 PM Kaxil Naik <[email protected]> wrote: > > Hi Dheeraj, > > +1 on the need but I'd look at this more broadly than assets. Most of the > webhooks I'd want Airflow to receive aren't just about data changing: > > - A GitHub PR gets the "deploy-preview" label: run the preview Dag for that > PR, and cancel the run if the PR is closed before it finishes. > - A task kicks off a dbt Cloud or Databricks job (or a day-long LLM batch) > and defers. Right now the trigger polls the job API until it's done. If the > service could call back instead, the task would just resume. > - Approving an HITL task from a Slack button. > - A Fivetran sync finished, so record an asset event. Your provider handles > this one well. > > Airflow can already do every one of these through the REST API. They don't > work today for the reason you called out: the sender can't hold a token. So > I'd like the core piece to be verified inbound events that can drive any of > these (asset event, new Dag run, resuming a deferred task, HITL response), > rather than asset events only. > > Resuming a task or cancelling a run also means the event has to point at > one specific run, and routing by asset name can't do that. Step Functions > solves it by giving each waiting task its own callback token. Something > similar here would also mean a leaked URL only affects one task, instead of > everything behind a shared secret. > > One more thing on assets: a run picks up every event queued since the > previous run, so ten PRs labeled within a minute could end up as one run > with ten events in triggering_asset_events. I don't think people would > expect that from a webhook. > > Thanks, > Kaxil > > On Thu, 15 Oct 2026 at 05:17, Dheeraj Turaga <[email protected]> wrote: > > > Hi all, > > > > I want to gauge the interest in supporting inbound webhooks for Airflow. > > > > Why? > > > > Webhooks are how most services announce events: source control and CI > > (GitHub, GitLab, Bitbucket), data tools (dbt Cloud, Fivetran, Airbyte), > > ML platforms, payment and commerce systems, incident tools (PagerDuty, > > Datadog) and issue trackers (Jira, Linear). Those events are often what > > a Dag is waiting for, but Airflow can't receive them: senders can't > > attach an Airflow API token, they sign deliveries with a shared secret, > > and each has its own payload format. So we poll with sensors, which hold > > capacity and react late, or run a relay service in front of Airflow. > > AssetWatcher covers pull-based queues; there's no push equivalent. > > > > Webhooks would let a Dag start the moment a dbt Cloud job finishes, a > > service is deployed, a PagerDuty incident opens or a vendor's export is > > ready. > > > > The idea: > > > > Turn incoming webhooks into asset events that trigger your Airflow Dags! > > > > Each verified delivery is recorded as an asset event named after what > > happened, and Dags subscribe by scheduling on that name. When a pull > > request in my-org/web gets the deploy-preview label, this Dag runs, with > > the payload in its triggering_asset_events: > > > > > > > > @dag(schedule=Asset("github/my-org/web/pull_request/labeled/deploy-preview")) > > > > Broader names work too: github/my-org/*/push/main fires on a push to > > main in any repository of the org. Senders don't need to know Dag IDs, > > one delivery can start several Dags, and the scheduler creates the runs > > as for any asset event. > > > > Try it out: > > > > I've built this as a provider for Airflow 3.1.1+, with GitHub, GitLab > > and Jira built in, plus a generic option for any service that signs its > > body with HMAC-SHA256 or sends a secret header: > > > > https://github.com/dheerajturaga/airflow-provider-inbound-webhooks > > https://pypi.org/project/airflow-provider-inbound-webhooks/ > > > > The README has a quick start that needs no third-party account, and > > docs/design.md covers what could move to core. > > > > If there's interest, I'm happy to draft an AIP. Thoughts welcome! > > > > Thanks, > > Dheeraj > >
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
