+1 on having a field to restore backwards compatible on dag level if the dag parameter is being changed. Most of our workloads involve submitting jobs to Spark and other upstream systems and each user has a corresponding pool. With deferred being not counted as active users had issues where many submissions were done that the upstream cannot handle. So for those users the pool was updated. There are other workloads like http based defers where users just poll and don't need to worry about the upstream capacity.
I guess deferred was initially documented such that pool slot is released and there can be more concurrent workloads. The definition of being counted as active depends on the workload and use case. It will be helpful to have the option to have this behaviour as optional and opt-in to avoid confusion. Thanks Regards, Karthikeyan S On Mon, Feb 23, 2026, 9:43 PM Vikram Koka via dev <[email protected]> wrote: > I definitely agree with the intent, but concerned about the actual > implications of making this change from a user experience perspective. > > With respect to pools, I would like an updated perspective on how useful > and used this is today. For example, I suspect that the async Python > operator change coming in the new AIP as part of 3.2 does not respect the > pools configuration either. > > The max active task configurations are very useful while using the Celery > executor, which is the majority today. I got a bunch of questions around > this as part of the backfill enhancements in 3.0. > > I hesitate making changes to these configuration options without clear > understanding and articulation of the tradeoffs. > > Just my two cents, > Vikram > > On Mon, Feb 23, 2026 at 2:34 AM Wei Lee <[email protected]> wrote: > > > I like what Jarek suggested, but we should avoid using the term > "Running". > > From Airflow's perspective, a Deferred task is not considered a Running > > task, even though it may be viewed differently in the user's context. > > > > Additionally, we are currently using the term "Executing" here > > > https://github.com/apache/airflow/blob/e0cd6e246c288d33f359ec2268b3d342832e9648/airflow-core/src/airflow/utils/state.py#L67 > > > > Maybe we can count Deferred and Running tasks as "Executing"? The thing > > that kinda bugs me is that "Defered" is also an IntermediateTIState here. > > > > On 2026/02/22 20:22:45 Natanel wrote: > > > Hello Jens, I agree with everything you said, for some reason the > > > "Deferred" state is not counted towards an active task, where > intuitively > > > it should be part of the group. > > > > > > As I see it at least, all the configurations talk about *active* tasks > > > (such as max_active_tasks, max_active_tis_per_dag, > > > max_active_tis_per_dagrun), which I think is a quite confusing term. > > > And to solve this, a clear explanation of what is an "active" task > should > > > be defined. > > > > > > It is possible to define that an "active" task is any task which is > > either > > > running, queued OR deferred, but this will enforce a new configuration > > for > > > backwards compatibility, such as "count_deffered_as_active" (yet this > is > > > more enforcing and global approach, which we might not want), while not > > > introducing too much additional complexity, as adding more parameters > by > > > which we schedule tasks will only make scheduling decisions harder, as > > more > > > parameters need to be checked, which will most likely slow down each > > > decision, and might slow down the scheduler. > > > > > > I liked Jarek's approach, however, I think that maybe instead of > > > introducing a few new params, we instead rename the current parameters, > > > while keeping behavior as is, slowly deprecating the "active" > > > configurations, as Jarek said, and for some time keep both the "active" > > and > > > the "running" param, while having the "active" be an alias for > "running" > > > until the "active" is deprecated. > > > > > > If there is a need for a param for deferred tasks, it is possible to > add > > > only for deferrable tasks, in order to not impact current scheduling > > > decisions done by the scheduler. > > > > > > I see both approaches as viable, yet I think that adding an additional > > > param might introduce more complexity, and maybe should be split out of > > the > > > regular task flow, as a deferrable task is not the same as a running > > task, > > > I tend to lean towards the first approach, as it seems to be the > > simplest, > > > however, the second approach might be more beneficial long-term. > > > > > > Best Regards, > > > Natanel > > > > > > On Sun, 22 Feb 2026 at 18:43, Jarek Potiuk <[email protected]> wrote: > > > > > > > +1. But I think that there are cases where people wanted to > > > > **actually** use `max_*` to limit how many workers the DAG or DAG run > > > > will take. So possibly we should give them such an option—for > example, > > > > max_running_tis_per_dag, etc. > > > > > > > > There is also the question of backward compatibility. I can see the > > > > possibility of side effects - if that changes "suddenly" after an > > > > upgrade. For example it might mean that some Dags will suddenly start > > > > using far fewer workers than before and become starved. > > > > > > > > So - if we want to change it, I think we should deprecate "_active" > > > > and possibly add two new sets of parameters with different names—but > > > > naming in this case is hard (more than usual). > > > > > > > > J. > > > > > > > > On Sun, Feb 22, 2026 at 5:25 PM Pavankumar Gopidesu > > > > <[email protected]> wrote: > > > > > > > > > > Hi Jens, > > > > > > > > > > Thanks for starting this discussion. I agree that we should update > > how > > > > > these tasks are counted. > > > > > > > > > > I previously started a PR[1] to include deferred tasks in > > > > max_active_tasks, > > > > > but I was sidetracked by other priorities. As you noted, this > change > > > > needs > > > > > to encompass not only max_active_tasks but also the other > parameters > > you > > > > > described. > > > > > > > > > > [1]: https://github.com/apache/airflow/pull/41560 > > > > > > > > > > Regards, > > > > > Pavan > > > > > > > > > > On Sun, Feb 22, 2026 at 12:43 PM constance.astronomer.io via dev < > > > > > [email protected]> wrote: > > > > > > > > > > > Agreed. In my opinion, the only time we should not be counting > > deferred > > > > > > tasks are for configurations that control worker slots, like > > number of > > > > > > tasks that run concurrently on a celery worker, since tasks in a > > > > deferred > > > > > > state are not running on a worker (although you can argue that a > > > > triggerer > > > > > > is a special kind of worker but I digress). > > > > > > > > > > > > For the examples you’ve listed, deferred tasks should be part of > > the > > > > > > equation since the task IS running, just not in a traditional > > worker. > > > > > > > > > > > > Thanks for bringing this up! This has been bothering me for > awhile. > > > > > > > > > > > > Constance > > > > > > > > > > > > > On Feb 22, 2026, at 4:18 AM, Jens Scheffler < > [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > Hi There! > > > > > > > > > > > > > > TLDR: In fix PR https://github.com/apache/airflow/pull/61769 > we > > > > came to > > > > > > the point that it seems today in Airflow Core the "Deferred" > state > > > > seems to > > > > > > be counted inconsistently. I would propose to consistently count > > > > "Deferred" > > > > > > into the counts of "Running". > > > > > > > > > > > > > > Details: > > > > > > > > > > > > > > * In Pools for a longer time (since PR > > > > > > > https://github.com/apache/airflow/pull/32709) it is possible > > to > > > > > > > decide whether tasks in deferred state are counted into pool > > > > > > > allocation or not. > > > > > > > * Before that Deferred were not counted into, which caused > tasks > > > > being > > > > > > > in deferred potentially overwhelm backends which defesated > the > > > > > > > purpose of pools > > > > > > > * Recently it was also seen that other limits we usually have > on > > Dags > > > > > > > defined as following do not consistently include deferred > into > > > > limits. > > > > > > > o max_active_tasks - `The number of task instances allowed > > to run > > > > > > > concurrently` > > > > > > > o max_active_tis_per_dag - `When set, a task will be able > to > > > > limit > > > > > > > the concurrent runs across logical_dates.` > > > > > > > o max_active_tis_per_dagrun - `When set, a task will be > able > > to > > > > > > > limit the concurrent task instances per Dag run.` > > > > > > > * This means at the moment defining a task as async/deferred > > escapes > > > > > > > the limits > > > > > > > > > > > > > > Code references: > > > > > > > > > > > > > > * Counting tasks in Scheduler on main: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L190 > > > > > > > * EXECUTION_STATES used for counting: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/ti_deps/dependencies_states.py#L21 > > > > > > > o Here "Deferred" is missing! > > > > > > > > > > > > > > Alternatives that I see: > > > > > > > > > > > > > > * Fix it in Scheduler consistently that limits are applied > > counting > > > > > > > Deferred always in > > > > > > > * There might be a historic reason that Deferred is not > counting > > in - > > > > > > > then a proper documentation would be needed - but I'd assume > > this > > > > > > > un-likely > > > > > > > * There are different opinions - then the behavior might need > to > > be > > > > > > > configurable. (But personally I can not see a reason for > having > > > > > > > deferred escaping the limits defined) > > > > > > > > > > > > > > Jens > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
