Hi Pradeepta, Thanks for the nice proposal. I am curious, how does the current autoscaler as well as the newly proposed custom evaluator plugin work for batch execution mode?
Regards Venkata krishnan On Thu, Mar 20, 2025 at 11:33 PM Pradeepta Choudhury <pchoudhur...@apple.com.invalid> wrote: > Hi Max, Good morning! Thanks so much for your review and feedback! > > I agree on proposal being generic and might not benefit a normal user > without having a decent understanding of how autoscaler’s logic works. > I had actually planned to contribute some predictive implementations using > the custom evaluator plugin. That said, having some built-in commonly used > predictive logic as part of the autoscaler sounds like a great idea. We can > probably introduce a scaling metric to denote predicted metric for > evaluated metrics to increase transparency that way?! I’ll work on a > follow-up for this. Thanks! > > Thanks > Pradeepta > > > > On 20 Mar 2025, at 10:36 PM, Maximilian Michels <m...@apache.org> wrote: > > > > Hi Pradeepta, > > > > Generally, the proposal looks good to me. The only issue I see is that > > it is fairly generic. On the one hand, it allows plugging in > > additional evaluation logic into the autoscaler, which allows for many > > new ways to extend the current autoscaling logic. On the other hand, > > normal users won't benefit from this. I wish we could build some > > predictive logic directly into the autoscaler. So far, we refrained > > from doing so because we wanted to get the basics right first > > (something that most Flink autoscaling solutions never got around to). > > > > +1 for the proposal as it is meaningful. Hopefully, we can also make > > some progress towards a built-in predictive logic as part of a new > > FLIP :) > > > > -Max > > > > On Thu, Mar 20, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> wrote: > >> > >> Hi! > >> > >> I think this is a very nice proposal, +1 from me! > >> > >> The only practical challenge is to actually be able to identify what job > >> vertex is what in the job graph to make good predictions / feed in > external > >> information but this is definitely out of scope here. > >> The autoscaler in general needs to have a good way to associate user > code / > >> operators with vertex ids for these types of use-cases that we need to > >> investigate as a followup. > >> > >> Cheers > >> Gyula > >> > >> On Wed, Mar 12, 2025 at 6:24 PM Pradeepta Choudhury > >> <pchoudhur...@apple.com.invalid> wrote: > >> > >>> Hi All, > >>> > >>> I would like to start a discussion of FLIP-514: Custom Evaluator plugin > >>> for Flink Autoscaler [1]. > >>> > >>> Feel free to share your thoughts and suggestions to make this feature > >>> better. > >>> > >>> > >>> [1] > >>> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-514*3A*Custom*Evaluator*plugin*for*Flink*Autoscaler__;JSsrKysrKw!!IKRxdwAv5BmarQ!a8jmPJ4UU5fJGp6O_sfSJJZ83bAz23j3xUcWGIFnnmw1qsSC51WvTx8hWy0zDa1S7Y4Ceesdfd1tsYzhP-vFSlVHZNiaihur$ > >>> > >>> > >>> > >>> Thanks > >>> Pradeepta > >
