Thanks Daniel for putting this together. As we are discussing here about pool/slots occupancy, at present there is no way to control how many deferred task can be executed except pool configuration IMHO.
There is issue created by Raphaelauv to control the number of deferred tasks on dag level. https://github.com/apache/airflow/issues/40528 I have created a pr to include this ask. https://github.com/apache/airflow/pull/41560. So posting this one to include it in discussion, might be useful for modeling decisions on the ongoing work, whether dag level config required or not for differed tasks. Regards, Pavan On Wed, Oct 9, 2024, 23:07 Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > and in case it wasn't clear, part of what i'm proposing is remove the "pool > can be configured to include deferred tasks" configuration option, because > we should just alays count a deferred task as taking up a pool slot and if > user doesn't want it to be part of pool during deferral just don't give it > pool. > > On Wed, Oct 9, 2024 at 3:04 PM Daniel Standish < > daniel.stand...@astronomer.io> wrote: > > > Some time ago, we added the ability to optionally configure a pool so > that > > it would count a deferred task as taking up a pool slot. > > > > It strikes me that there isn't much point in specifying a pool for a > > deferrable task if the task will not take up a slot when it is deferred. > > > > why? > > > > well the point of deferrable task is, the task spends most of > > it's lifecycle in deferred state. > > > > is it useful to have it take up a slot at the first and last couple > > seconds of its lifecycle? methinks no. > > > > seems to me for 3.0 we should just declare that tasks that have a pool > > will occupy a pool slot for the whole "running" lifecycle (which may > > include task execution and deferral). > > > > and ideally it would occupy the slot continuously (and not vacate the > slot > > e.g. when the task momentarily becomes "scheduled" after coming out of > > deferral. maybe we need better precision without states to do this but > > let's think about what is ideal for the moment. > > > > DISCUSS > > > > >
