Hi Max, Good morning! Thanks so much for your review and feedback! I agree on proposal being generic and might not benefit a normal user without having a decent understanding of how autoscaler’s logic works. I had actually planned to contribute some predictive implementations using the custom evaluator plugin. That said, having some built-in commonly used predictive logic as part of the autoscaler sounds like a great idea. We can probably introduce a scaling metric to denote predicted metric for evaluated metrics to increase transparency that way?! I’ll work on a follow-up for this. Thanks!
Thanks Pradeepta > On 20 Mar 2025, at 10:36 PM, Maximilian Michels <m...@apache.org> wrote: > > Hi Pradeepta, > > Generally, the proposal looks good to me. The only issue I see is that > it is fairly generic. On the one hand, it allows plugging in > additional evaluation logic into the autoscaler, which allows for many > new ways to extend the current autoscaling logic. On the other hand, > normal users won't benefit from this. I wish we could build some > predictive logic directly into the autoscaler. So far, we refrained > from doing so because we wanted to get the basics right first > (something that most Flink autoscaling solutions never got around to). > > +1 for the proposal as it is meaningful. Hopefully, we can also make > some progress towards a built-in predictive logic as part of a new > FLIP :) > > -Max > > On Thu, Mar 20, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> wrote: >> >> Hi! >> >> I think this is a very nice proposal, +1 from me! >> >> The only practical challenge is to actually be able to identify what job >> vertex is what in the job graph to make good predictions / feed in external >> information but this is definitely out of scope here. >> The autoscaler in general needs to have a good way to associate user code / >> operators with vertex ids for these types of use-cases that we need to >> investigate as a followup. >> >> Cheers >> Gyula >> >> On Wed, Mar 12, 2025 at 6:24 PM Pradeepta Choudhury >> <pchoudhur...@apple.com.invalid> wrote: >> >>> Hi All, >>> >>> I would like to start a discussion of FLIP-514: Custom Evaluator plugin >>> for Flink Autoscaler [1]. >>> >>> Feel free to share your thoughts and suggestions to make this feature >>> better. >>> >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler >>> >>> >>> >>> Thanks >>> Pradeepta
