+1 for this proposal, it allows advanced Users to introduce some specific strategies.
Also, if some strategies are general enough, it's better to put them inside the autoscaler. Best, Rui On Sat, Mar 29, 2025 at 3:09 PM <gyula.f...@gmail.com> wrote: > Hi Venkata, > > Let’s keep this discussion about the FLIP and not other unrelated topics. > > You are welcome to post the question in a new thread . > > Thanks > Gyula > > Sent from my iPhone > > > On 29 Mar 2025, at 02:58, Venkatakrishnan Sowrirajan <vsowr...@asu.edu> > wrote: > > > > Hi Pradeepta, > > > > Thanks for the nice proposal. I am curious, how does the current > autoscaler > > as well as the newly proposed custom evaluator plugin work for batch > > execution mode? > > > > Regards > > Venkata krishnan > > > > > >> On Thu, Mar 20, 2025 at 11:33 PM Pradeepta Choudhury > >> <pchoudhur...@apple.com.invalid> wrote: > >> > >> Hi Max, Good morning! Thanks so much for your review and feedback! > >> > >> I agree on proposal being generic and might not benefit a normal user > >> without having a decent understanding of how autoscaler’s logic works. > >> I had actually planned to contribute some predictive implementations > using > >> the custom evaluator plugin. That said, having some built-in commonly > used > >> predictive logic as part of the autoscaler sounds like a great idea. We > can > >> probably introduce a scaling metric to denote predicted metric for > >> evaluated metrics to increase transparency that way?! I’ll work on a > >> follow-up for this. Thanks! > >> > >> Thanks > >> Pradeepta > >> > >> > >>>> On 20 Mar 2025, at 10:36 PM, Maximilian Michels <m...@apache.org> > wrote: > >>> > >>> Hi Pradeepta, > >>> > >>> Generally, the proposal looks good to me. The only issue I see is that > >>> it is fairly generic. On the one hand, it allows plugging in > >>> additional evaluation logic into the autoscaler, which allows for many > >>> new ways to extend the current autoscaling logic. On the other hand, > >>> normal users won't benefit from this. I wish we could build some > >>> predictive logic directly into the autoscaler. So far, we refrained > >>> from doing so because we wanted to get the basics right first > >>> (something that most Flink autoscaling solutions never got around to). > >>> > >>> +1 for the proposal as it is meaningful. Hopefully, we can also make > >>> some progress towards a built-in predictive logic as part of a new > >>> FLIP :) > >>> > >>> -Max > >>> > >>> On Thu, Mar 20, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> > wrote: > >>>> > >>>> Hi! > >>>> > >>>> I think this is a very nice proposal, +1 from me! > >>>> > >>>> The only practical challenge is to actually be able to identify what > job > >>>> vertex is what in the job graph to make good predictions / feed in > >> external > >>>> information but this is definitely out of scope here. > >>>> The autoscaler in general needs to have a good way to associate user > >> code / > >>>> operators with vertex ids for these types of use-cases that we need to > >>>> investigate as a followup. > >>>> > >>>> Cheers > >>>> Gyula > >>>> > >>>> On Wed, Mar 12, 2025 at 6:24 PM Pradeepta Choudhury > >>>> <pchoudhur...@apple.com.invalid> wrote: > >>>> > >>>>> Hi All, > >>>>> > >>>>> I would like to start a discussion of FLIP-514: Custom Evaluator > plugin > >>>>> for Flink Autoscaler [1]. > >>>>> > >>>>> Feel free to share your thoughts and suggestions to make this feature > >>>>> better. > >>>>> > >>>>> > >>>>> [1] > >>>>> > >> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-514*3A*Custom*Evaluator*plugin*for*Flink*Autoscaler__;JSsrKysrKw!!IKRxdwAv5BmarQ!a8jmPJ4UU5fJGp6O_sfSJJZ83bAz23j3xUcWGIFnnmw1qsSC51WvTx8hWy0zDa1S7Y4Ceesdfd1tsYzhP-vFSlVHZNiaihur$ > >>>>> > >>>>> > >>>>> > >>>>> Thanks > >>>>> Pradeepta > >> > >> >
