> > I’m wondering why do we want to remove this. The design seems to be > reasonable, but yep, it might not be as helpful as mentioned. > > > is it useful to have it take up a slot at the first and last couple > seconds > of its lifecycle? methinks no. > > Some edge cases “might” be helpful. I guess? A deferrable operator that’s > not (or could not) implemented well probably need it. My main question is > whether this configuration is causing confusion or complicating logic in > the code base? Otherwise, we probably could just keep it? >
Ok so why... Airflow is complicated and in some cases provides too many configuration choices to user, or too-confusing configuration choices. Airflow 3 is the time to clean these things up. If we only *add* configurations and features and params, we can see where that goes. Just increasing complexity, which makes airflow more confusing and frustrating to use, and harder to maintain. There is a strong impulse to "just make it configurable", when considering one behavior or another. We need to fight this impulse. And to be fair sometimes it's done to introduce a behavior change or improvement in a backward compatible way. And that may be in part what's going on here. But now it's 3.0 and we can make the change. I think this option on pools is an instance where, we should just decide that tasks count against the pool for the full "running phase" of the task (incl on worker and triggerer and in between). It's not like this param, if left on pools, will eat your children. But I am saying I don't really think it makes sense there. So I would be in favor of simplifying it and removing it, and making a good choice for the user, and reducing the complexity in airflow just a little bit.
