Hey All! The main purpose of the adaptive scheduler is to be able to adapt to changing resource availability and requirements. Originally it was designed to work based on resource availability (with reactive scaling) so when we have more resources we scale up, if we have less scale down, at that point the question of releasing TMs was not a problem.
Now with the resource requirements api added somewhat recently the adaptive scheduler actually adapts to job resource requirements too. This works well when increasing the requirements, however when decreasing the requirements the TMs are not released in many cases as highlighted in this thread. I don't necessarily want to label it as a bug but I feel that the behaviour definitely does not align with the original intention of the adaptive scheduler and how it works in other cases. I think we should definitely fix this in 1.X line too given how important this feature is, and this behaviour can actually cause a lot of problems for some users. 1.X will be around for quite some time still. Cheers, Gyula On Wed, Nov 6, 2024 at 7:32 AM Yuepeng Pan <panyuep...@apache.org> wrote: > Hi, community. > > > When working on ticket[1] we have received some lively discussions and > valuable > feedback[2](thanks for Matthias, Rui, Gyula, Maximilian, Tison, etc.), the > main issues are that: > > 1. > > When the job runs in an application cluster, could the default > behavior of AdaptiveScheduler not actively releasing Taskmanagers > resources during downscaling be considered a bug? > 2. > > If so,should we fix it in flink 1.x? > > > I’d like to start a discussion to hear more comments about it to define > the next step and I have sorted out some information in the doc[3] > regarding this discussion for you. > > Looking forward to your comments and attention. > > Thank you. > > Best, > Yuepeng Pan > > [1] https://issues.apache.org/jira/browse/FLINK-33977 > > [2] https://github.com/apache/flink/pull/25218#issuecomment-2401913141 > [3] > https://docs.google.com/document/d/1Rwwl2aGVz9g5kUJFMP5GMlJwzEO_a-eo4gPf7gITpdw/edit?tab=t.0#heading=h.s4i4hehbbli5 > > >
