Yeah... Off-by one... Good eye - I lied too :) (noticed it after I sent the message. I wish email had the ability to correct typos)..
2) -> yes we agree, but to clarify a bit - we need it at OPERATOR level not TASK level. The difference comes from who defines it should be the Operator's Author not the DAG Author. I.e. we should be able to define "use_pool_when_deferred" (working name) when you define the operator, not when you create the operator as a task in DAG. So basically IMHO it should have the ability to internally set this property of the BaseOperator, but it's not necessary to expose it via the `__init__` method of the actual CustomDeferrableOperator(BaseOperator). We still CAN expose it via __init__, but I'd not say it's desired. Example: 1) example 1. RunMyGPUFineTuningOperator. Pool = num shared GPU, The operator does: a) wait in deferrable for a MODEL to appear b) upload the model and fine-tunes it (non-deferrable, uses GPU). "use_pool_when_deferred" = False 2) example 2. UpdateNewSalesforceUsersOperator. Pool = num salesforce connections. (Protect Salesforce API from being overloaded - our licence has only 10 parallel connections possible) The operator does a) checks if new users are defined (by polling Salesforce API) - deferred b) updates the user with new fields via Salesforce API. "use_pool_when_deferred" = True Enough. J. On Sun, Oct 20, 2024 at 4:45 PM Daniel Standish <daniel.stand...@astronomer.io.invalid> wrote: > So yeah hopefully we all agree that if we keep it, we should move it to > task. > > I guess we can think of this thread as two items: > > 1. if we keep ability to have tasks not occupy a pool slot, shouldn't it > be configured at task level? I think so. > 2. but should we keep the ability to have tasks not be considered at > task level? > 3. if tasks are to stay in pools when deferred, ideally they should do > so continuously (e.g. including when in between worker and triggerer) > > > Ok I lied, three items. But 3 is more like a reminder that there is this > bad behavior :) > > Anyway, let's move on to focus on number 2, whether we should provide users > a configuration option to make tasks "drop out" of the pool when deferred. > > After reading your message Jarek, I did not come away with an understanding > of a practical use case for having the task vacate the pool slot when > deferred. Can you offer an example or two? > > > > On Sun, Oct 20, 2024 at 7:29 AM Daniel Standish < > daniel.stand...@astronomer.io> wrote: > > > Totally agree that if we *must* make it configurable, then at task is the > > right place. Will read the rest of it later :) >
