Kind reminder to everyone who still wants to add comments If no further comments i think we can move this to a vote.
On Tue, May 5, 2026 at 12:00 AM Jarek Potiuk <[email protected]> wrote: > Very interesting - also will take a look shortly - and great how it seems > to tap into SLA urgency indeed.. > > On Mon, May 4, 2026 at 10:08 PM Przemysław Mirowski <[email protected]> > wrote: > > > +1 for Weighted Aging and SLA Urgency proposition. > > ________________________________ > > From: Elad Kalif <[email protected]> > > Sent: 30 April 2026 12:08 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] AIP-100: Eliminate Scheduler Starvation > > > > I love the idea of dynamic priorities and I think this is a good > direction. > > > > On Thu, Apr 30, 2026 at 12:58 PM Ash Berlin-Taylor <[email protected]> > wrote: > > > > > Thanks, this re-drafted version looks interesting. I’m trying to > > > internalise and understand the new proposal. I’l leave a few > > > comments/questions on the confluence page as I go. > > > > > > (I have to say, the move away from epic stored procedure def is a > welcome > > > change!) > > > > > > -ash > > > > > > > On 22 Apr 2026, at 14:45, Natanel <[email protected]> wrote: > > > > > > > > Hello community, I want to spark again a discussion that was held in > > the > > > > past and delayed (due to the focus on releasing airflow 3.2), now > that > > > 3.2 > > > > is released, I think it might be a good Idea to bring up the > discussion > > > > again. > > > > > > > > Wiki: > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-100+Eliminate+Scheduler+Starvation+On+Concurrency+Limits > > > > > > > > In the current situation, tasks may starve in airflow, and in large > > scale > > > > deployments (hundreds of thousands of tasks (or more) per day) we > tend > > to > > > > experience severe starvation, where a group of tasks may starve other > > > > tasks, not allowing them to run, as described in the wiki. > > > > > > > > After the february devcall, where I have proposed the AIP, a few > > comments > > > > have arised, and so I had begun to research again about different > > > > scheduling algorithms and I have added to the considerations as part > of > > > the > > > > AIP. > > > > > > > > As of now, the state of the AIP is where there are a few ideas > proposed > > > > (some of which are pretty similar to each other, while others are > quite > > > > different), as the main concern from the devcall was that the > > approaches > > > > given might not be the best way to solve the issue, as it is a very > > hard > > > > problem to solve. > > > > > > > > After that, I have made some edits to the AIP and to the > propositions, > > in > > > > order to help decide and clarify the advantages and disadvantages of > > each > > > > approach. > > > > The current "best approach" can be found here here > > > > < > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Currentbestproposition > > > >, > > > > where the new proposed algorithms are defined here > > > > < > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618462#AIP100EliminateSchedulerStarvationOnConcurrencyLimits-Othernonagingalgorithmsother_algs > > > > > > > > . > > > > > > > > In order to continue with the effort, a community consensus needs to > be > > > > reached about the preferred solution/solutions, once this is done, it > > is > > > > possible to go on and implement + stress test the proposed > solution/s. > > > > > > > > I would appreciate a review from community members, moreover, I would > > > also > > > > appreciate any new propositions or improvements which can be done. > > > > > > > > Best regards > > > > Natanel. > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
