Hello, I've got a question but I can't find a way to add a comment under the AIP.
The 'Weighted Aging and SLA Urgency' solves the starvation issue but do we have an idea of what happens with the scheduler's performance? Christos On Thu, May 14, 2026 at 7:47 AM Elad Kalif <[email protected]> wrote: > Kind reminder to everyone who still wants to add comments > If no further comments i think we can move this to a vote. > > On Tue, May 5, 2026 at 12:00 AM Jarek Potiuk <[email protected]> wrote: > > > Very interesting - also will take a look shortly - and great how it seems > > to tap into SLA urgency indeed.. > > > > On Mon, May 4, 2026 at 10:08 PM Przemysław Mirowski <[email protected]> > > wrote: > > > > > +1 for Weighted Aging and SLA Urgency proposition. > > > ________________________________ > > > From: Elad Kalif <[email protected]> > > > Sent: 30 April 2026 12:08 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] AIP-100: Eliminate Scheduler Starvation > > > > > > I love the idea of dynamic priorities and I think this is a good > > direction. > > > > > > On Thu, Apr 30, 2026 at 12:58 PM Ash Berlin-Taylor <[email protected]> > > wrote: > > > > > > > Thanks, this re-drafted version looks interesting. I’m trying to > > > > internalise and understand the new proposal. I’l leave a few > > > > comments/questions on the confluence page as I go. > > > > > > > > (I have to say, the move away from epic stored procedure def is a > > welcome > > > > change!) > > > > > > > > -ash > > > > > > > > > On 22 Apr 2026, at 14:45, Natanel <[email protected]> wrote: > > > > > > > > > > Hello community, I want to spark again a discussion that was held > in > > > the > > > > > past and delayed (due to the focus on releasing airflow 3.2), now > > that > > > > 3.2 > > > > > is released, I think it might be a good Idea to bring up the > > discussion > > > > > again. > > > > > > > > > > Wiki: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > > > > > > > > > In the current situation, tasks may starve in airflow, and in large > > > scale > > > > > deployments (hundreds of thousands of tasks (or more) per day) we > > tend > > > to > > > > > experience severe starvation, where a group of tasks may starve > other > > > > > tasks, not allowing them to run, as described in the wiki. > > > > > > > > > > After the february devcall, where I have proposed the AIP, a few > > > comments > > > > > have arised, and so I had begun to research again about different > > > > > scheduling algorithms and I have added to the considerations as > part > > of > > > > the > > > > > AIP. > > > > > > > > > > As of now, the state of the AIP is where there are a few ideas > > proposed > > > > > (some of which are pretty similar to each other, while others are > > quite > > > > > different), as the main concern from the devcall was that the > > > approaches > > > > > given might not be the best way to solve the issue, as it is a very > > > hard > > > > > problem to solve. > > > > > > > > > > After that, I have made some edits to the AIP and to the > > propositions, > > > in > > > > > order to help decide and clarify the advantages and disadvantages > of > > > each > > > > > approach. > > > > > The current "best approach" can be found here here > > > > > < > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition > > > > >, > > > > > where the new proposed algorithms are defined here > > > > > < > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs > > > > > > > > > > . > > > > > > > > > > In order to continue with the effort, a community consensus needs > to > > be > > > > > reached about the preferred solution/solutions, once this is done, > it > > > is > > > > > possible to go on and implement + stress test the proposed > > solution/s. > > > > > > > > > > I would appreciate a review from community members, moreover, I > would > > > > also > > > > > appreciate any new propositions or improvements which can be done. > > > > > > > > > > Best regards > > > > > Natanel. > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
