Hi Vikram, Natanel has reworked the approach, and it is receiving a much warmer reception than before. This is Natanel's restart attempt, and there has been no other discussion about it. While the "why" remains the same—addressing starvation issues experienced by various users (including Jens' team)—the previous proposals were rejected by Ash, Jens, and me due to excessive complexity, such as building new algorithms via stored procedures and such.
The current proposal is simpler and introduces an interesting way to prioritize SLA callbacks - whichis an interesting feature to have. While it requires further scrutiny, it appears to be a direction we could potentially accept. Best, Jarek On Thu, May 14, 2026 at 4:50 PM Vikram Koka via dev <[email protected]> wrote: > I probably missed the updates here, but when this was brought up at the dev > call a while ago, I thought the response was a firm "No". > Did I miss the follow-on updates? > > The discussion centered on the "why". > This is such a "power user" feature, which is fine, but it lacks > articulation of the projected benefit in quantifiable terms. > > It has very high technical complexity in the core of Airflow, which could > cause massive ripple effects and therefore requires a cautious approach. > > I left a comment in the AIP document as well, about needing to understand > the "why" better. > > Vikram > > > > On Thu, May 14, 2026 at 1:28 AM Christos Bisias <[email protected]> > wrote: > > > Hello, > > > > I've got a question but I can't find a way to add a comment under the > AIP. > > > > The 'Weighted Aging and SLA Urgency' solves the starvation issue but do > we > > have an idea of what happens with the scheduler's performance? > > > > Christos > > > > On Thu, May 14, 2026 at 7:47 AM Elad Kalif <[email protected]> wrote: > > > > > Kind reminder to everyone who still wants to add comments > > > If no further comments i think we can move this to a vote. > > > > > > On Tue, May 5, 2026 at 12:00 AM Jarek Potiuk <[email protected]> wrote: > > > > > > > Very interesting - also will take a look shortly - and great how it > > seems > > > > to tap into SLA urgency indeed.. > > > > > > > > On Mon, May 4, 2026 at 10:08 PM Przemysław Mirowski < > [email protected] > > > > > > > wrote: > > > > > > > > > +1 for Weighted Aging and SLA Urgency proposition. > > > > > ________________________________ > > > > > From: Elad Kalif <[email protected]> > > > > > Sent: 30 April 2026 12:08 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] AIP-100: Eliminate Scheduler Starvation > > > > > > > > > > I love the idea of dynamic priorities and I think this is a good > > > > direction. > > > > > > > > > > On Thu, Apr 30, 2026 at 12:58 PM Ash Berlin-Taylor <[email protected] > > > > > > wrote: > > > > > > > > > > > Thanks, this re-drafted version looks interesting. I’m trying to > > > > > > internalise and understand the new proposal. I’l leave a few > > > > > > comments/questions on the confluence page as I go. > > > > > > > > > > > > (I have to say, the move away from epic stored procedure def is a > > > > welcome > > > > > > change!) > > > > > > > > > > > > -ash > > > > > > > > > > > > > On 22 Apr 2026, at 14:45, Natanel <[email protected]> > > wrote: > > > > > > > > > > > > > > Hello community, I want to spark again a discussion that was > held > > > in > > > > > the > > > > > > > past and delayed (due to the focus on releasing airflow 3.2), > now > > > > that > > > > > > 3.2 > > > > > > > is released, I think it might be a good Idea to bring up the > > > > discussion > > > > > > > again. > > > > > > > > > > > > > > Wiki: > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > > > > > > > > > > > > > In the current situation, tasks may starve in airflow, and in > > large > > > > > scale > > > > > > > deployments (hundreds of thousands of tasks (or more) per day) > we > > > > tend > > > > > to > > > > > > > experience severe starvation, where a group of tasks may starve > > > other > > > > > > > tasks, not allowing them to run, as described in the wiki. > > > > > > > > > > > > > > After the february devcall, where I have proposed the AIP, a > few > > > > > comments > > > > > > > have arised, and so I had begun to research again about > different > > > > > > > scheduling algorithms and I have added to the considerations as > > > part > > > > of > > > > > > the > > > > > > > AIP. > > > > > > > > > > > > > > As of now, the state of the AIP is where there are a few ideas > > > > proposed > > > > > > > (some of which are pretty similar to each other, while others > are > > > > quite > > > > > > > different), as the main concern from the devcall was that the > > > > > approaches > > > > > > > given might not be the best way to solve the issue, as it is a > > very > > > > > hard > > > > > > > problem to solve. > > > > > > > > > > > > > > After that, I have made some edits to the AIP and to the > > > > propositions, > > > > > in > > > > > > > order to help decide and clarify the advantages and > disadvantages > > > of > > > > > each > > > > > > > approach. > > > > > > > The current "best approach" can be found here here > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition > > > > > > >, > > > > > > > where the new proposed algorithms are defined here > > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs > > > > > > > > > > > > > > . > > > > > > > > > > > > > > In order to continue with the effort, a community consensus > needs > > > to > > > > be > > > > > > > reached about the preferred solution/solutions, once this is > > done, > > > it > > > > > is > > > > > > > possible to go on and implement + stress test the proposed > > > > solution/s. > > > > > > > > > > > > > > I would appreciate a review from community members, moreover, I > > > would > > > > > > also > > > > > > > appreciate any new propositions or improvements which can be > > done. > > > > > > > > > > > > > > Best regards > > > > > > > Natanel. > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > >
