and in case it wasn't clear, part of what i'm proposing is remove the "pool can be configured to include deferred tasks" configuration option, because we should just alays count a deferred task as taking up a pool slot and if user doesn't want it to be part of pool during deferral just don't give it pool.
On Wed, Oct 9, 2024 at 3:04 PM Daniel Standish < daniel.stand...@astronomer.io> wrote: > Some time ago, we added the ability to optionally configure a pool so that > it would count a deferred task as taking up a pool slot. > > It strikes me that there isn't much point in specifying a pool for a > deferrable task if the task will not take up a slot when it is deferred. > > why? > > well the point of deferrable task is, the task spends most of > it's lifecycle in deferred state. > > is it useful to have it take up a slot at the first and last couple > seconds of its lifecycle? methinks no. > > seems to me for 3.0 we should just declare that tasks that have a pool > will occupy a pool slot for the whole "running" lifecycle (which may > include task execution and deferral). > > and ideally it would occupy the slot continuously (and not vacate the slot > e.g. when the task momentarily becomes "scheduled" after coming out of > deferral. maybe we need better precision without states to do this but > let's think about what is ideal for the moment. > > DISCUSS > >
