+1. Not batching. I think we should generally deprecate / remove the auto-batching behavior as it is impossible to predict or control. If we introduce batching, it should be configurable: specifying the number of messages, the batching time windows etc. Unpredictable behaviour like this is dangerous, especially if users start relying on it and it suddenly gets impacted by unrelated things. In this case the big issue is that many "noisy neighbour" cases can change the behaviour in unpredictable ways. You might not even see the batching happening regularly - if events are "slow-ish," - but suddenly you might suddenly start working when you trigger a backfill for a completely different DAG, for example. That's a recipe for disaster.
On Sat, Sep 12, 2026 at 1:20 PM Jarek Potiuk <[email protected]> wrote: > NOTE: To Jake and others posting to devlist - as not everyone might be > aware—please subscribe to devlist before posting (see "community" in > https://airflow.apache.org). > > Otherwise we have to moderate your posts (only subscribers can write > without moderation), and you might not see the answer when someone replies > to "dev@". > > J. > > On Fri, Sep 11, 2026 at 6:55 PM Jake McGrath via dev < > [email protected]> wrote: > >> I agree with Constance that this is something that needs a bit more >> discussion; this is something that I’ve bumped into a good bit lately, >> specifically, with Asset-watching of S3. >> >> I like to think about this starting with the DAG. If my DAG has Tasks >> within it that are only meant to parse a single file, then a single run >> for >> > 1 Asset Event would most likely break my DAG/not be properly handled. >> >> I do think there is a use-case for the existing “batching” sort of >> behavior. However, I’d personally lean towards NOT batching Asset Events >> by default. >> >> >> >> >> On Sep 10, 2026 at 7:40:29 PM, Constance Martineau via dev < >> [email protected]> wrote: >> >> > Hi all, >> > >> > Raising something that has been coming up over the past few years, >> > especially recently with Asset Watchers: When a Dag is asset-triggered, >> the >> > scheduler consumes every pending AssetEvent for that Dag in one loop and >> > creates a single DagRun for all of them >> (_create_dag_runs_asset_triggered). >> > If five files land and generate five events before the next scheduler >> loop >> > runs, you get one DagRun that consumes all five events, not five >> separate >> > DagRuns. Most users assume 1 event -> 1 DagRun and read the batchings as >> > Airflow dropping or missing events, and how much gets batched depends on >> > scheduler loop and Dag parallelism settings, not anything declared in >> the >> > Dag (see https://github.com/apache/airflow/issues/56750). This >> behavior is >> > intentional, dating back to when datasets shipped in 2.4, but the >> > perception that "this is a bug" is real. This issue has come up as a >> > specific point at Airflow Summit talks two years running, with speakers >> > assuming this was Airflow 2 flakiness that had since been fixed. >> > >> > It's not just a perception problem, either. Any asset-triggered Dag that >> > isn't explicitly written to iterate `dag_run.consumed_asset_events` will >> > silently under-process when multiple events land in the same run. It >> looks >> > like "1 run = 1 file" and quietly drops the rest. This isn't a >> hypothetical >> > scenario. The exact use-case that prompted this is that someone in one >> of >> > our client facing teams is building an S3 trigger on the AssetWatcher >> > framework (there's no official S3 event trigger for this yet, the only >> > documented AssetWatcher pattern today is SQS) that emits one event per >> > updated file, with several files landing in the same scheduler loop. >> > Because how many events get bundled depends on scheduler cadence and >> > parallelism rather than anything declared in the Dag, this behavior is >> also >> > untestable. There's no way to write a CI test that reliably asserts "N >> > events produce N runs". >> > >> > There's already community momentum here: >> > >> > - #55956 <https://github.com/apache/airflow/issues/55956> proposes a >> > `max_asset_events` param, milestoned for 3.4.0 >> > - #56750 >> > < >> > >> https://medium.com/@MarinAgli1/a-look-into-airflow-data-aware-scheduling-and-dynamic-task-mapping-8c548d4ad79 >> > > >> > groups the related issues (#53896 >> > <https://github.com/apache/airflow/issues/53896>, #56691 >> > <https://github.com/apache/airflow/issues/56691>, #56050 >> > <https://github.com/apache/airflow/issues/56050> and #47398 >> > <https://github.com/apache/airflow/issues/47398>) and proposes a >> > Dag-level toggle (`asset_grouping`) rather than an all-or-nothing >> global >> > switch. >> > >> > Regarding the question of whether this can only be a global config: I >> don't >> > think so. `_create_dag_runs_asset_triggered` already loops per-Dag and >> only >> > fetches that Dag's own pending events before building its DagRun, so a >> > per-Dag opt-in is localized to that branch and shouldn't require >> touching >> > the shared code path every other asset-scheduled Dag depends on. I'd >> model >> > this the same way we already handle `catchup`: a Dag-level parameter >> (like >> > `asset_grouping`) that falls back to a global >> `asset_grouping_by_default` >> > in airflow configs when unset, mirroring `catchup` / >> `catchup_by_default`. >> > That gives Dag authors an explicit override where usage is genuinely >> mixed, >> > while still letting an org flip the behaviour fleet-wide for every Dag >> that >> > hasn't opted in, without touching Dag code. SCrocky's issue for example >> > Idescribes running both patterns side by side in the same deployment. >> > >> > Where I want actual discussion: What should `asset_grouping_by_default` >> > ship as? Every proposal so far (including ours) assumes it has to >> default >> > to today's batched behaviour, to avoid a breaking change. I want to make >> > the case for defaulting it to `False` (1 event -> 1 DagRun) instead, >> > because the cost asymmetric. >> > >> > - If we default to unbatched and someone was relying on batching, the >> > worst case is an extra DagRun. They were never able to control the >> batch >> > size to begin with as it depends on the scheduler cadence and >> > parallelism, >> > not anything declared in the Dag, so their code already has to >> tolerate a >> > variable number of events per run. An occasional extra run is just >> more >> > of >> > the same variance they already had to handle, not a new failure mode. >> > - If we keep batching as the default, everyone who wants per-event >> > semantics, which judging by the summit and this thread is most >> people's >> > mental model, has to actively work around it. They need to inspect how >> > many >> > events landed in a run and add conditional/expansion logic to split >> them >> > back out, which is exactly the workaround we're discussing internally >> > right >> > now. This imposes real, ongoing complexity on the majority to protect >> a >> > minority's default. >> > >> > I don't think this is a close call. >> > >> > I'm not trying to let perfect be the enemy of good. If we can only get >> > consensus on an opt-in with today's default, that's still a real >> > improvement, but I'd rather we make the case for the better default >> before >> > settling for that. >> > >> > Curious what others thing, especially anyone closer to #56750 >> > < >> > >> https://medium.com/@MarinAgli1/a-look-into-airflow-data-aware-scheduling-and-dynamic-task-mapping-8c548d4ad79 >> > > >> > . >> > >> > Thanks, >> > Constance >> > >> > >> > -- >> > >> > Constance Martineau >> > >> > Staff Product Manager >> > >> > Email: [email protected] >> > >> > Time zone: US Eastern (EST UTC-5 / EDT UTC-4) >> > >> >
